# Deploy on VMs and cloud

> Plan, configure, secure and create a HotKV cluster across several hosts: sizing, systemd, ports and firewalls, passwords, announce settings and failure detection.

URL: https://docs.hotkv.com/clustering/production/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

This page takes you from empty hosts to a running, secured cluster. The
examples use two hosts, `10.0.0.1` (`node1.example.com`) and `10.0.0.2`
(`node2.example.com`), with three nodes on each on ports 7000 to 7002, so the
commands can be recorded and copied. Two hosts are enough to learn on, but
not to run on: see [Plan the layout](#plan-the-layout).

If you have not built a cluster before, do the
[quick start](https://docs.hotkv.com/clustering/quick-start/index.md) first. For encrypting traffic, add
[Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md) after this page.

## Plan the layout

| Decision | Recommendation |
| --- | --- |
| Primaries | At least 3. Add primaries when data or writes outgrow the ones you have. |
| Replicas | At least 1 per primary. A primary without a replica cannot fail over. |
| Hosts | At least 3, with each primary and its replicas on different hosts (and, in a cloud, different availability zones). |
| Nodes per host | One where you can. Several are fine as long as a primary and its own replica never share a host. |
| Ports | Client port, for example 7000, and the bus port 10000 above it. Plan both. |
| Memory | Set [`maxmemory`](https://docs.hotkv.com/reference/configuration/index.md#maxmemory) below the machine's memory and leave room for replication buffers. |

Why three hosts and not two: the cluster stops serving when the nodes that
fail leave fewer than a majority of the primaries (see
[Slots and redirects](https://docs.hotkv.com/clustering/index.md#slots-and-redirects)). With three primaries
and two hosts, one host carries two primaries. If it fails, the one remaining
primary is on the minority side, and the whole cluster answers `CLUSTERDOWN`
until the host is back. With three hosts, any one host can fail and a
majority is left. The same reasoning applies across availability zones.

## Prepare the hosts

### Open files

Each client connection uses a file descriptor. HotKV raises its own limit at
startup up to the hard limit, so that `maxclients` (10000 by default) fit.
These are the limits of an ordinary login on a Linux host, the soft limit
first and the hard limit second:

```bash
ulimit -n; ulimit -Hn
```

```text
1024
524288
```

If the hard limit is too low, HotKV lowers `maxclients` to what fits and says
so in its log (this node was started with a hard limit of 1024):

```text
The open files limit only allows 960 clients, so maxclients is lowered from 10000 to 960. Raise it with 'ulimit -n' or LimitNOFILE= to allow more.
```

Under systemd, `LimitNOFILE=65535` in the unit raises the limit for the
process. The unit below sets it, and the running node shows it:

```text
Max open files            65535                65535                files
```

(That is the `Max open files` line of `/proc/<pid>/limits`.)

### Clocks

Keep every node's clock in sync with NTP or chrony. A time to live on an AI
entry is turned into a fixed deadline on the primary, and each node judges
that deadline by its own clock, so a replica whose clock is ahead drops
entries early. See [AI engines in a cluster](https://docs.hotkv.com/clustering/ai-engines/index.md).

### Install

Install HotKV on every host as described in
[Install HotKV](https://docs.hotkv.com/get-started/install/index.md). The packages create the `hotkv`
user and directories and a service for one server. A host that runs one node
can use that service, with the settings below in `/etc/hotkv/hotkv.conf`. A
host that runs several nodes needs one service per node, as shown next.

## Configure a node

Every node needs cluster mode on, its own data directory, a password, and
addresses that other hosts can reach. This is the file for the first node on
the first host, `/etc/hotkv/7000.conf`:

```text
bind 10.0.0.1 127.0.0.1
port 7000
protected-mode yes
requirepass change-me
masterauth change-me
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
appendonly yes
appendfsync everysec
dir /var/lib/hotkv/7000
logfile /var/log/hotkv/7000.log
```

| Setting | Why it is there |
| --- | --- |
| `bind` | The addresses the node listens on, here its private address and loopback. The cluster bus listens on the same addresses. A node bound to `127.0.0.1` only cannot be reached by the other hosts. |
| `requirepass` | The password of every client, replica and cluster tool. Use the same value on all nodes. See [Authentication and protected mode](#authentication-and-protected-mode). |
| `masterauth` | The same password again, for the node's own logins to other nodes: a replica's login, and the login of an atomic slot migration. It is not needed for replication when `requirepass` is the same everywhere, but set it so that `CLUSTER MIGRATESLOTS` works. |
| `cluster-enabled` | Cluster mode. It cannot be changed at runtime. |
| `cluster-config-file` | Where the node keeps what it knows about the cluster. A name without a directory is placed inside `dir`; an absolute path is used as it is. |
| `cluster-node-timeout` | How long a node may be silent before the others flag it. 15000 ms is the default, written here to make it visible. See [Failure detection](#failure-detection). |
| `appendonly`, `appendfsync` | Write every change to an append-only file, flushed once a second. See [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md). |
| `dir` | The node's data directory. Never share one between nodes. |

The same node in `hotkv.toml` (use one format per node):

```toml
[network]
bind = "10.0.0.1 127.0.0.1"
port = 7000
protected_mode = true

[security]
requirepass = "change-me"

[cluster]
enabled = true
config_file = "nodes.conf"
node_timeout = 15000

[persistence]
dir = "/var/lib/hotkv/7000"
appendonly = true
appendfsync = "everysec"

[general]
logfile = "/var/log/hotkv/7000.log"
```

`masterauth` has no key in `hotkv.toml`, so a node configured with TOML cannot
hold it in its file: use a `hotkv.conf` for such nodes, or run `CONFIG SET
masterauth` after the start (that does not survive a restart).

The nodes on one host differ by port and directory, and the nodes on another
host by `bind`. [Cluster configuration](https://docs.hotkv.com/clustering/configuration/index.md) lists every
setting and its TOML name.

Write the file of the first node (as `root`, since it is under `/etc`), make
the files of the other two nodes on the host from it, create the data and log
directories, and keep the files, which hold the password, readable by `root`
and the `hotkv` group only:

```bash
for port in 7001 7002; do sudo sed s/7000/$port/g /etc/hotkv/7000.conf | sudo tee /etc/hotkv/$port.conf > /dev/null; done
sudo chgrp hotkv /etc/hotkv/700{0,1,2}.conf
sudo chmod 640 /etc/hotkv/700{0,1,2}.conf
sudo install -d -o hotkv -g hotkv /var/lib/hotkv/7000 /var/lib/hotkv/7001 /var/lib/hotkv/7002 /var/log/hotkv
sudo ls -l /etc/hotkv
```

```text
-rw-r----- 1 root hotkv 247 Jan 15 10:00 7000.conf
-rw-r----- 1 root hotkv 247 Jan 15 10:00 7001.conf
-rw-r----- 1 root hotkv 247 Jan 15 10:00 7002.conf
```

(Do the same on the second host with `bind 10.0.0.2 127.0.0.1`. The commands
below that write under `/etc`, `/var/lib` or `/var/log` need `sudo` too.)

Replicas need no extra setting. They are made by `redis-cli --cluster create`
(or `CLUSTER REPLICATE`) and copy from their primary's client port with the
same password.

## Run nodes under systemd

A unit template runs one node per port. This one is based on the packaged
service, with the port as the instance name, so `hotkv-node@7000` reads
`/etc/hotkv/7000.conf`:

```ini
[Unit]
Description=HotKV cluster node on port %i
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/hotkv --config /etc/hotkv/%i.conf
TimeoutStartSec=infinity
TimeoutStopSec=infinity
Restart=on-failure
RestartSec=5
LimitNOFILE=65535
User=hotkv
Group=hotkv
WorkingDirectory=/var/lib/hotkv/%i
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true
ReadWritePaths=/var/lib/hotkv/%i /var/log/hotkv

[Install]
WantedBy=multi-user.target
```

Save it as `/etc/systemd/system/hotkv-node@.service`, create the directories
`/var/lib/hotkv/7000` and `/var/log/hotkv` owned by `hotkv`, and start the
node:

```bash
sudo systemctl daemon-reload
sudo systemctl start hotkv-node@7000 hotkv-node@7001
systemctl is-active hotkv-node@7000 hotkv-node@7001
```

```text
active
active
```

`sudo systemctl enable hotkv-node@7000` starts a node at boot. With
`Type=notify` the unit counts as started once the node is ready, and
`TimeoutStartSec=infinity` lets a node with a large data set load without
being killed. To use TOML for a node, add a drop-in that replaces the command:

```ini
[Service]
ExecStart=
ExecStart=/usr/bin/hotkv --config /etc/hotkv/%i.toml
```

Both formats work: the `.conf` file above under this unit, and the `.toml`
file above started with `hotkv --config /etc/hotkv/7000.toml`, each answering
`CLUSTER INFO` with `cluster_enabled:1`. `systemctl status` shows the state, and the log of a node
under systemd is in the journal:

```bash
systemctl status hotkv-node@7000 --no-pager | grep -E "hotkv-node|Active|Status"
```

```text
● hotkv-node@7000.service - HotKV cluster node on port 7000
     Loaded: loaded (/etc/systemd/system/hotkv-node@.service; disabled; preset: enabled)
     Active: active (running) since Thu 2026-01-15 10:00:00 UTC; 2s ago
     Status: "Ready to accept connections"
```

A node that is killed (for example by the kernel) is started again five
seconds later. A restarted node loads its `nodes.conf` and rejoins as
the node it was.

## Ports and firewalls

Every node needs these ports open to the right sources:

| Port | Used for | Open it to |
| --- | --- | --- |
| Client port (7000) | Clients, `redis-cli --cluster`, and replicas copying from their primary | Your application hosts, your admin hosts, and every other node |
| Bus port (client port + 10000, 17000) | Gossip, failure detection and elections | Every other node only. Never the internet, never application hosts. |
| TLS port, when you use TLS | Clients over TLS. With `tls-cluster yes` the bus port is this port + 10000. | As the client port and the bus port |

The nodes need to reach each other in both directions, because each node
opens connections to all the others. A firewall that allows only one direction
breaks the cluster.

In a cloud, express the same table with the provider's constructs. On
AWS, allow the client and bus port ranges from the security group itself
(a rule whose source is the group the nodes belong to), and from your
application's security group for the client port only. On Azure, use a network
security group with the `VirtualNetwork` service tag, or the subnet's address
range, as the source of the node-to-node rules, and the application subnet
for the client port. On Google Cloud, create VPC firewall rules whose targets
are a network tag on the cluster instances and whose sources are the same tag
(for the bus) and your application's tag (for clients). In every case, check
the result from a node: a port that is blocked times out instead of
answering.

On the hosts themselves, a firewall such as iptables or nftables needs
the same rules. The failure you see when the bus ports are blocked is
described next. The rule below drops the bus ports from the other host. Add it
on both hosts, and remove it afterwards:

```bash
sudo iptables -I INPUT -p tcp --dport 17000:17002 -s 10.0.0.1 -j DROP
```

(Run the rule on the second host with `-s` naming the first host, and on the
first host with `-s 10.0.0.2`.) Creating the cluster then never finishes:

```bash
export REDISCLI_AUTH=change-me; timeout 40 redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 \
  10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes; echo "exit status $?"
```

```text
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.2:7002 to 10.0.0.1:7000
Adding replica 10.0.0.1:7002 to 10.0.0.2:7000
Adding replica 10.0.0.2:7001 to 10.0.0.1:7001
...
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.......................................exit status 124
```

`redis-cli` waits at `Waiting for the cluster to join` and prints dots until
the nodes agree. A bus port that is blocked times out, and one that is open
answers, even though a plain client gets a reset:

```bash
redis-cli -h 10.0.0.2 -p 17000 -t 3 PING
```

```text
Could not connect to Redis at 10.0.0.2:17000: Connection timed out
```

After the rule is removed, the same command prints `Error: Connection reset
by peer`, which shows the port is reachable. The nodes do not join on their
own, and creating again is refused until the nodes are reset. The steps are
in [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md#create-waits-forever).

## Authentication and protected mode

A node that listens on a non-loopback address and has no password is in
protected mode, and serves only clients on its own host. Clients on other
hosts get an error that says what to do:

```bash
redis-cli -h 10.0.0.2 -p 7000 PING
```

```text
DENIED HotKV is running in protected mode because protected mode is on and the default user has no password: only clients on the loopback interface are served. To serve other hosts, set a password for the default user (requirepass), or turn protected mode off where the network already keeps strangers out: 'CONFIG SET protected-mode no' from the loopback interface (then 'CONFIG REWRITE' to keep it), 'protected-mode no' in the configuration file, or '--protected-mode no' on the command line.
```

The node warns about it when it starts:

```text
Protected mode is on and the default user has no password, so clients from other hosts are refused (DENIED). Set a password (requirepass), or protected-mode no where the network keeps strangers out.
```

The fix for a cluster is a password. Set the same `requirepass` on every
node. Replicas use it to prove themselves to their primary, so replication
needs nothing else. `masterauth` on the replicas does the same if their
`requirepass` differs. Set `masterauth` to the same password on every node as
well, as the file above does: `CLUSTER MIGRATESLOTS` and atomic resharding with
`valkey-cli` sign in to the target with `masterauth` (and `masteruser`), never
with `requirepass`, and fail with `NOAUTH` without it (see
[When a move fails](https://docs.hotkv.com/clustering/scaling/index.md#when-a-move-fails)). With the
password set, the other hosts get a clear error and can authenticate:

```text
$ redis-cli -h 10.0.0.2 -p 7000 PING
NOAUTH Authentication required.
$ REDISCLI_AUTH=wrong redis-cli -h 10.0.0.2 -p 7000 PING
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
NOAUTH Authentication required.
$ REDISCLI_AUTH=change-me redis-cli -h 10.0.0.2 -p 7000 PING
PONG
```

Give the password to `redis-cli` with the `REDISCLI_AUTH` environment
variable. The `-a` option works too but prints a warning, because the
password is visible in the process list. `redis-cli --cluster` needs the
password for every command, and the errors you get without it are in
[Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md#authentication-errors).

Turn protected mode off (`protected-mode no`) only where the network already
keeps everyone else out. Then no password is needed, and anyone who can reach
a client port can read and write.

Your applications should not log in with this administrator password. Give
them a user of their own once the cluster exists: see
[An ACL user for your applications](#an-acl-user-for-your-applications).

## Create the cluster

With all six nodes running and the ports open, one command forms the cluster.
`redis-cli` places each replica on a different host than its primary, as the
output shows:

```bash
REDISCLI_AUTH=change-me redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 \
  10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes
```

```text
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.2:7002 to 10.0.0.1:7000
Adding replica 10.0.0.1:7002 to 10.0.0.2:7000
Adding replica 10.0.0.2:7001 to 10.0.0.1:7001
M: 6e11c647ab79dd11819cc29dc690ab737b365c23 10.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: f25db3e9f70991572429580fd6fe79dfc8204e7c 10.0.0.1:7001
   slots:[10923-16383] (5461 slots) master
S: 65296b9f4bbef4dd5be6d2ca4084ee83c733e51f 10.0.0.1:7002
   replicates 67770cc41d2a95d2f1e0eeeed072575f792ad0af
M: 67770cc41d2a95d2f1e0eeeed072575f792ad0af 10.0.0.2:7000
   slots:[5461-10922] (5462 slots) master
S: 27f82ff2546a6b432bea748f8d0c711f4b64d226 10.0.0.2:7001
   replicates f25db3e9f70991572429580fd6fe79dfc8204e7c
S: 2e87f71486030ec02495d3c0fad932f13c7219ac 10.0.0.2:7002
   replicates 6e11c647ab79dd11819cc29dc690ab737b365c23
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 10.0.0.1:7000)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

The nodes on one host are the first three addresses, so they become the
primaries; every replica is on the other host. In this layout the first host
holds two primaries, which is the weakness described under
[Plan the layout](#plan-the-layout). With three or more hosts, list the
addresses so that the first three are on three different hosts. `valkey-cli`
takes the same command. The cluster needs a few seconds before it accepts
writes, so wait, check it, and try a write and a read through a node of the
other host:

```bash
until REDISCLI_AUTH=change-me redis-cli -p 7000 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done; echo ready
```

```text
ready
```

```bash
REDISCLI_AUTH=change-me redis-cli --cluster check 10.0.0.1:7000
```

```text
10.0.0.1:7000 (6e11c647...) -> 0 keys | 5461 slots | 1 slaves.
10.0.0.2:7000 (67770cc4...) -> 0 keys | 5462 slots | 1 slaves.
10.0.0.1:7001 (f25db3e9...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.1:7000)
...
[OK] All 16384 slots covered.
```

```bash
export REDISCLI_AUTH=change-me
redis-cli -c -h 10.0.0.1 -p 7000 SET greeting hello
redis-cli -c -h 10.0.0.2 -p 7002 GET greeting
```

```text
OK
hello
```

## An ACL user for your applications

Applications should not use the administrator password. Create a user with
only the commands they need, once the cluster exists. ACL users are per node:
a user created on one node does not exist on the others, and a client that
follows a redirect to a node without the user fails. Here the user is created
on one node and a key that lives on another is written:

```bash
export REDISCLI_AUTH=change-me
redis-cli -h 10.0.0.1 -p 7000 ACL SETUSER app on ">app-secret" "~*" "&*" +@all -@admin -@dangerous
REDISCLI_AUTH=app-secret redis-cli -c --user app -h 10.0.0.1 -p 7000 SET greeting hello
```

```text
OK
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
Error:
```

Create the user on every node, replicas included:

```bash
export REDISCLI_AUTH=change-me
for node in 10.0.0.1:7001 10.0.0.1:7002 10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002; do
  redis-cli -h ${node%:*} -p ${node#*:} ACL SETUSER app on ">app-secret" "~*" "&*" +@all -@admin -@dangerous
done
REDISCLI_AUTH=app-secret redis-cli -c --user app -h 10.0.0.1 -p 7000 SET greeting hello
REDISCLI_AUTH=app-secret redis-cli -c --user app -h 10.0.0.1 -p 7000 CONFIG GET maxclients
```

```text
OK
OK
OK
OK
OK
OK
NOPERM User app has no permissions to run the 'config|get' command
```

The user can write keys on every node and cannot run administrative
commands. To keep users across restarts without repeating this, give every
node the same ACL file with `aclfile` and copy it to every host. The file is
read at start, and `ACL SAVE` writes it back with the passwords as hashes.
Replicas, `CLUSTER MIGRATESLOTS` and the move of AI entries between nodes do
not read the ACL file. They use `requirepass` and `masterauth` from the node's
configuration, so keep both set to the same password as the `default` user has
in the file:

```text
user app on >app-secret ~* &* +@all -@admin -@dangerous
user default on >change-me ~* &* +@all
```

After a node starts with this file, `ACL WHOAMI` as `app` answers `app`,
and `ACL LIST` shows both users with their passwords stored as hashes. See
[ACL](https://docs.hotkv.com/operate/acl/index.md) for the rule syntax.

## Announce addresses and NAT

Each node tells the others, and its clients, an address to reach it. By
default that is the address it listens on. When clients or other nodes must
use a different one (a NAT, a load balancer, a container network, a name),
change what the node announces. All of these settings change at runtime with
`CONFIG SET`, and are also settings in the file. The examples use a
three-primary cluster on one machine (ports 7000 to 7002), so the addresses
are loopback ones, and the names are placeholders that do not resolve.

First, what a cluster prints by default. `CLUSTER SLOTS` gives each slot
range with the node's address:

```text
127.0.0.1:7000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7000
      3) "98d247471e61efcaf5bad3bfaf6b0d4df08166e3"
      4) (empty array)
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7001
      3) "94115e7410a2f14bf47f7d6946aea46340452335"
      4) (empty array)
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7002
      3) "d33906c39424f47a599ac9b415acc1d549e7526d"
      4) (empty array)
```

### Host names

`cluster-announce-hostname` gives a node a name, and
`cluster-preferred-endpoint-type hostname` makes redirects and the cluster
commands hand clients that name instead of the IP address. Use it when your
clients resolve names but IP addresses change (for example in Kubernetes or
behind DNS):

```bash
redis-cli -p 7000 CONFIG SET cluster-announce-hostname node1.example.com
redis-cli -p 7001 CONFIG SET cluster-announce-hostname node2.example.com
redis-cli -p 7002 CONFIG SET cluster-announce-hostname node3.example.com
redis-cli -p 7000 CONFIG SET cluster-preferred-endpoint-type hostname
```

```text
OK
OK
OK
OK
```

`CLUSTER SLOTS` now gives the name first and keeps the IP address in the
extra fields. `CLUSTER NODES` shows the name after the address, and a
redirect carries it:

```text
127.0.0.1:7000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "node1.example.com"
      2) (integer) 7000
      3) "98d247471e61efcaf5bad3bfaf6b0d4df08166e3"
      4) 1) "ip"
         2) "127.0.0.1"
...
127.0.0.1:7000> CLUSTER NODES
d33906c39424f47a599ac9b415acc1d549e7526d 127.0.0.1:7002@17002,node3.example.com master - 0 1791064588374 3 connected 10923-16383
94115e7410a2f14bf47f7d6946aea46340452335 127.0.0.1:7001@17001,node2.example.com master - 0 1791064587373 2 connected 5461-10922
98d247471e61efcaf5bad3bfaf6b0d4df08166e3 127.0.0.1:7000@17000,node1.example.com myself,master - 0 0 1 connected 0-5460
127.0.0.1:7000> GET greeting
(error) MOVED 12714 node3.example.com:7002
```

`CLUSTER SHARDS` lists the name as `hostname`, and as `endpoint` too while the
preferred type is `hostname`. A client then needs to resolve the name. Names
may contain letters, digits, `-` and `.`, and be shorter than 256 characters.

With `cluster-preferred-endpoint-type unknown-endpoint`, a node gives no
address at all and the client is expected to use the one it reached the node
at. Redirects carry only the port, and `CLUSTER SLOTS` shows a nil endpoint:

```text
$ redis-cli -p 7000 CONFIG SET cluster-preferred-endpoint-type unknown-endpoint
OK
127.0.0.1:7000> GET greeting
(error) MOVED 12714 :7002
127.0.0.1:7000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) (nil)
      2) (integer) 7000
      3) "98d247471e61efcaf5bad3bfaf6b0d4df08166e3"
      4) 1) "ip"
         2) "127.0.0.1"
         3) "hostname"
         4) "node1.example.com"
...
```

The default is `ip`. Set it back, because the examples that follow assume it:

```bash
redis-cli -p 7000 CONFIG SET cluster-preferred-endpoint-type ip
```

```text
OK
```

### A NAT or load balancer in front of a node

When clients reach a node at an address other than the one the nodes use
among themselves, set the address and port that clients are given. Nodes keep
using the real address and the bus port among themselves. This example tells
clients that the node on 7002 is at `203.0.113.10:17379`, an address from the
documentation range:

```bash
redis-cli -p 7002 CONFIG SET cluster-announce-client-ipv4 203.0.113.10
redis-cli -p 7002 CONFIG SET cluster-announce-client-port 17379
```

```text
OK
OK
```

The redirect and `CLUSTER NODES` carry the client address, and the bus port
(`@17002`) stays as it was:

```text
127.0.0.1:7000> GET greeting
(error) MOVED 12714 203.0.113.10:17379
127.0.0.1:7000> CLUSTER NODES
d33906c39424f47a599ac9b415acc1d549e7526d 203.0.113.10:17379@17002,node3.example.com master - 0 1791064592379 3 connected 10923-16383
94115e7410a2f14bf47f7d6946aea46340452335 127.0.0.1:7001@17001,node2.example.com master - 0 1791064591378 2 connected 5461-10922
98d247471e61efcaf5bad3bfaf6b0d4df08166e3 127.0.0.1:7000@17000,node1.example.com myself,master - 0 0 1 connected 0-5460
```

`cluster-announce-client-ipv6` and `cluster-announce-client-tls-port` do the
same for IPv6 clients and for TLS clients. A value that is not a valid address
is refused:

```text
$ redis-cli -p 7002 CONFIG SET cluster-announce-client-ipv4 999.1.1.1
ERR CONFIG SET failed (possibly related to argument 'cluster-announce-client-ipv4') - Invalid IPv4 address
```

Clear the two settings again before the next example, by setting the address to
`""` and the port to `0`:

```bash
redis-cli -p 7002 CONFIG SET cluster-announce-client-ipv4 ""
redis-cli -p 7002 CONFIG SET cluster-announce-client-port 0
```

```text
OK
OK
```

### The address the nodes use among themselves

`cluster-announce-ip` changes the address the node gives the other nodes (and
clients, unless `cluster-announce-client-ipv4` is set). It takes an IP address
or a name that the other nodes can resolve. This example announces `localhost`:

```text
$ redis-cli -p 7001 CONFIG SET cluster-announce-ip localhost
OK
127.0.0.1:7000> CLUSTER NODES
d33906c39424f47a599ac9b415acc1d549e7526d 127.0.0.1:7002@17002,node3.example.com master - 0 1791064595384 3 connected 10923-16383
94115e7410a2f14bf47f7d6946aea46340452335 localhost:7001@17001,node2.example.com master - 0 1791064594383 2 connected 5461-10922
98d247471e61efcaf5bad3bfaf6b0d4df08166e3 127.0.0.1:7000@17000,node1.example.com myself,master - 0 0 1 connected 0-5460
```

An empty value goes back to the address the node binds to (or the one the
others see its connections come from, when it binds to every address).
`cluster-announce-port` and `cluster-announce-bus-port` change the client and
bus port the same way, for a port mapping. With `cluster-announce-port 17380`
the redirect for that node read `MOVED 12714 127.0.0.1:17380`. Only announce a
bus port that actually reaches the node's bus, or the other nodes cannot
connect to it.

### The availability zone

`availability-zone` is a name for where the node runs. It is part of `HELLO`
and `INFO server` for clients, and other nodes list it in `CLUSTER SLOTS` and
`CLUSTER SHARDS`, so a client can prefer a replica in its own zone. Set it on
each node, at runtime or in the file:

```bash
redis-cli -p 7002 CONFIG SET availability-zone zone-a
```

```text
OK
```

Then the other nodes list it:

```text
127.0.0.1:7000> CLUSTER SLOTS
...
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7002
      3) "d33906c39424f47a599ac9b415acc1d549e7526d"
      4) 1) "hostname"
         2) "node3.example.com"
         3) "availability-zone"
         4) "zone-a"
127.0.0.1:7002> HELLO 3
1# "server" => "hotkv"
2# "version" => "8.4.0"
3# "proto" => (integer) 3
4# "id" => (integer) 10
5# "mode" => "cluster"
6# "role" => "master"
7# "modules" => (empty array)
8# "availability_zone" => "zone-a"
```

```bash
redis-cli -p 7002 INFO server | grep availability
```

```text
availability_zone:zone-a
```

The version in `HELLO` is the Redis version HotKV reports
(`compat-redis-version`, 8.4.0 by default), not the HotKV version.

## Failure detection

A node flags a peer as possibly failing after `cluster-node-timeout`
milliseconds without an answer, and marks it failed once a majority of the
primaries agree. A replica of a failed primary then holds an election, and the
cluster serves the slots again when it wins. The time that takes is the node
timeout plus the time to agree and vote.

Choose the timeout by what a pause means for you: a shorter timeout fails over
faster, and makes a slow network or a busy host more likely to be taken for a
failure. Do not go below a value that your slowest host and network can always
meet. The log lines of a failover are in
[Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md#failover-did-not-happen).

`cluster-require-full-coverage` (default `yes`) decides what happens when a
slot has no owner, for example after a primary and all its replicas are lost.
With `yes` the whole cluster stops serving: every node answers `CLUSTERDOWN
The cluster is down`. With `no`, nodes keep serving the slots that are still
covered, and a client that asks for a key in a lost slot is redirected to the
failed node and sees a connection error:

```text
$ redis-cli -c -p 7001 GET greeting
CLUSTERDOWN The cluster is down
...
$ redis-cli -c -p 7001 GET b
Could not connect to Redis at 127.0.0.1:7000: Connection refused
```

(The first command ran with `yes`, the second with `no`, after the shard that
owns `b` was lost.) Use `yes` when partial data is worse than none. Use `no`
when your data splits cleanly and applications can handle errors for part of
the key space. Set it to the same value on every node.

## Next steps

- [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md): encrypt clients, the bus and replication.
- [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md): protect the data.
- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md): replicas, manual
  failover and what a failover loses.
- [Production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md): check everything
  before you go live.