Cluster troubleshooting

Not on HotKV CloudOn-Premise: no license needed

Start with the symptom you see. Each section ends with a way to confirm that the fix worked. Log lines are shown without the date, time and level that start each line in the log. Two habits save time: run redis-cli --cluster check against any node first, and read the log of the node that reports the problem (see Where the logs are).

Symptom	Likely cause	Section
redis-cli --cluster create prints dots after Waiting for the cluster to join and never ends	The bus ports between the nodes are blocked	Create waits forever
[ERR] Node ... is not empty	The node already knows other nodes, owns slots or holds keys	A node is not empty
DENIED HotKV is running in protected mode ...	No password, and the client is not on the same host	DENIED
NOAUTH or WRONGPASS from redis-cli, or AUTH failed after a redirect	Wrong or missing password, or a user that exists on some nodes only	Authentication errors
A replica shows master_link_status:down	The replica and primary passwords differ	A replica cannot sync
CLUSTERDOWN	A slot is uncovered, or the node is on the minority side	CLUSTERDOWN
MOVED to an address the client cannot reach	The node announces an address clients cannot use	Redirected to an unreachable address
MOVED shown by a client that does not follow it	The client is not cluster aware	MOVED and ASK shown to your client
CROSSSLOT	A command names keys in different slots	CROSSSLOT
TRYAGAIN	A multi-key command hit a slot that is moving	TRYAGAIN
LOADING HotKV is loading the dataset in memory on a replica	The replica is copying its primary’s data	LOADING on a replica
A node exits at start	Port in use, a damaged nodes.conf, a bad setting	A node does not start
redis-cli --cluster check reports [ERR] or [WARNING]	Nodes disagree, slots are uncovered or open	Nodes disagree
A primary died and no replica took over	nofailover, no majority, or a replica that never synced	Failover did not happen
Nodes keep flagging each other as failing	cluster-node-timeout is too low for the network	Flapping failures
NOREPLICAS on CLUSTER SETSLOT	A replica is connected but not acknowledging	NOREPLICAS
A slot migration seems stuck	A slot is left open on a node	A stuck migration
TLS handshake errors in the log	Wrong CA, expired certificate, plain node in a TLS cluster	TLS handshake failures

Create waits forever

>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.......................................

redis-cli --cluster create introduced the nodes to each other with CLUSTER MEET and now waits until they all see each other. That happens over the cluster bus, not over the client port, so if even one pair of nodes cannot open a connection to each other’s bus port (client port + 10000), the dots never stop. The client ports are fine, which is why the earlier steps work. With the bus ports blocked by a firewall rule (see Ports and firewalls) the command never ends; timeout 40 makes it end:

export REDISCLI_AUTH=change-me; timeout 40 redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes; echo "exit status $?"

...
Waiting for the cluster to join
.......................................exit status 124

Check from one host whether another host’s bus port answers. A blocked port times out, and a reachable one answers at once (a plain client does not speak the bus protocol, so the answer is a reset):

redis-cli -h 10.0.0.2 -p 17000 -t 3 PING

Could not connect to Redis at 10.0.0.2:17000: Connection timed out

With the firewall rule removed, the same command prints:

Error: Connection reset by peer

On the first node, CLUSTER NODES shows only the nodes on its own side, and CLUSTER INFO reports cluster_state:fail because slots are missing:

$ REDISCLI_AUTH=change-me redis-cli -p 7000 CLUSTER NODES
aca663cab2beed27d99f8c1efecf2f07fe43805d 10.0.0.1:7002@17002 master - 0 1780066857039 3 connected
28efd03d876eef35d833762acdb3f5ec7a1df046 10.0.0.1:7001@17001 master - 0 1780066856038 2 connected 10923-16383
ce65054c3ecabefa6336daecec2a1b5ab5947f9d 10.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460
$ REDISCLI_AUTH=change-me redis-cli -p 7000 CLUSTER INFO | head -n 6
cluster_enabled:1
cluster_state:fail
cluster_slots_assigned:10922
cluster_slots_ok:10922
cluster_slots_pfail:0
cluster_slots_fail:0

Fix. Open the bus ports between all nodes, both ways. See the firewall rules. A cluster whose nodes announce an address the others cannot reach behaves the same, see Redirected to an unreachable address.

Opening the ports does not complete the interrupted create. The nodes were told about each other once and gave up, so waiting does not help (CLUSTER NODES keeps showing only the nodes on its own side), and creating again fails with [ERR] Node ... is not empty. Reset the nodes and create again, as shown in the next section.

Confirm. The create finishes with [OK] All 16384 slots covered. and redis-cli --cluster check reports no errors.

A node is not empty

[ERR] Node 10.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

redis-cli --cluster create only accepts fresh nodes: no known peers, no slots and no keys. The usual causes are an earlier failed create, a node reused from another cluster, or a node whose old nodes.conf was kept.

Fix. If you do not need the data on these nodes, empty and reset every node, then create again. CLUSTER RESET HARD gives the node a new ID and forgets the others. A primary that holds keys refuses to reset, so run FLUSHALL first:

export REDISCLI_AUTH=change-me
for node in 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002; do
  redis-cli -h ${node%:*} -p ${node#*:} CLUSTER RESET HARD
done

OK
OK
OK
OK
OK
OK

To keep the data of a node, add it to a running cluster as a new node instead of creating from it. See Scaling.

Confirm. redis-cli --cluster create runs through.

DENIED, protected mode

redis-cli -h 10.0.0.2 -p 7000 PING

DENIED HotKV is running in protected mode because protected mode is on and the default user has no password: only clients on the loopback interface are served. To serve other hosts, set a password for the default user (requirepass), or turn protected mode off where the network already keeps strangers out: 'CONFIG SET protected-mode no' from the loopback interface (then 'CONFIG REWRITE' to keep it), 'protected-mode no' in the configuration file, or '--protected-mode no' on the command line.

The node listens on a non-loopback address, its default user has no password, and protected mode is on (the default), so it serves only clients on the same host. The same command run on the node’s own host answers PONG. At startup the node logs a warning, and it logs each client it refuses:

Protected mode is on and the default user has no password, so clients from other hosts are refused (DENIED). Set a password (requirepass), or protected-mode no where the network keeps strangers out.
Refused a client from 10.0.0.1:58824: protected mode is on and the default user has no password, so only clients on a loopback address are served

Because nodes also reach each other’s client port for replication and resharding, this blocks replicas and redis-cli --cluster as well.

Fix. Set the same requirepass on every node, the right fix on any network you do not fully control. Where the network already keeps strangers out, you can turn protected mode off instead. See Authentication and protected mode.

Confirm. The same command from the other host answers NOAUTH Authentication required. once a password is set, and PONG when you give the password (or after you turn protected mode off).

Authentication errors

With a password set, redis-cli --cluster needs it for every node, and a missing or wrong password stops it at the first node:

redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes

[ERR] Node 10.0.0.1:7000 NOAUTH Authentication required.

REDISCLI_AUTH=wrong redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes

Node 10.0.0.1:7000 replied with error:
WRONGPASS invalid username-password pair or user is disabled.

Fix. Give the password through the REDISCLI_AUTH environment variable (or -a, which prints a warning that the password is visible in the process list). Every node must accept it, so use the same requirepass everywhere.

A user that exists on some nodes only. ACL users are per node. A client logged in as an application user on one node is redirected to a node that does not know the user, and redis-cli -c stops:

REDISCLI_AUTH=app-secret redis-cli -c --user app -h 10.0.0.1 -p 7000 SET greeting hello

AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
Error:

Keys that live on the node where the user exists work, and the others fail, which makes the problem look random. Fix. Create the same user on every node, primaries and replicas (see Authentication and protected mode).

Confirm. A write to a key on each node succeeds with the application user.

A replica cannot sync

A replica whose password differs from its primary’s never finishes its first copy. On the replica:

$ redis-cli -p 7451 INFO replication | grep -E 'role|master_link_status|master_port'
role:slave
master_port:7450
master_link_status:down

The replica’s log says what its primary refused, and the primary’s log shows the other end:

Replication: cannot connect to primary 127.0.0.1:7450: it refused this node's replication password (attempt 1, retrying)
Replication: refused replica 127.0.0.1:53216: it does not have this node's replication password (replication.password, else requirepass)

Cause. Replicas prove they know the primary’s replication password. That is replication.password if set, else masterauth, else requirepass. Fix. Use the same requirepass on every node, or set masterauth on the replicas to the primary’s password. Both work; the change takes effect without a restart.

Confirm. master_link_status:up on the replica.

CLUSTERDOWN

CLUSTERDOWN has three forms. Which one you see tells you the cause.

CLUSTERDOWN Hash slot not served. No node owns the slot of the key: the slot is unassigned. That happens after CLUSTER DELSLOTS, after CLUSTER FORGET of the node that owned the slot, or on a cluster whose slots were never assigned. A shard that is only down still owns its slots, so its keys do not get this error: they get The cluster is down, or a redirect to the dead node when cluster-require-full-coverage is no, as described below.

127.0.0.1:7200> GET greeting
(error) CLUSTERDOWN Hash slot not served

(Slot 12714, the slot of greeting, had been removed with CLUSTER DELSLOTS 12714 on every node. A key in a slot that still has an owner gets The cluster is down instead, described next, while the cluster state is fail.)

Find the uncovered slots with redis-cli --cluster check, and see Nodes disagree to fix them.

CLUSTERDOWN The cluster is down. The node’s cluster state is fail, and it refuses every command that names a key. There are two reasons, and the log says which:

Cluster: the cluster is down: At least one hash slot is not served by any available node, and cluster-require-full-coverage is yes
Cluster: the cluster is down: this node is in a minority partition, reaching no majority of the primaries that serve slots

• A slot is uncovered and cluster-require-full-coverage is yes (the default). One lost shard stops the whole cluster. If you prefer to keep serving the slots that are covered, set cluster-require-full-coverage no on every node. Keys in the lost shard’s slots are then redirected to the failed node, and a client sees Could not connect to Redis at 127.0.0.1:7430: Connection refused.
• This node is on the minority side. It cannot reach a majority of the primaries that own slots, for example after a network split, or after two of three primaries stopped:

127.0.0.1:7200> GET greeting
(error) CLUSTERDOWN The cluster is down
127.0.0.1:7200> SET greeting hi
(error) CLUSTERDOWN The cluster is down

Restore the missing nodes or the network. The state returns to ok on its own when the node can see a majority again. A primary that has just been restarted also stays in fail for a few seconds before it accepts writes.

CLUSTERDOWN The cluster is down and only accepts read commands. With cluster-allow-reads-when-down yes, a node in the fail state still serves reads and answers writes with this error.

Confirm. CLUSTER INFO reports cluster_state:ok and a write to a key in the affected slot succeeds.

Redirected to an unreachable address

A client that follows a redirect stalls, then fails:

$ redis-cli -c -t 3 -p 7420 SET greeting hello
Could not connect to Redis at 172.31.0.5:7422: Connection timed out
$ redis-cli -p 7420 GET greeting
MOVED 12714 172.31.0.5:7422

The first command works against the node you contacted, but the redirect names an address that the client cannot reach from where it runs: the node’s address inside a container network, a private address behind a NAT or a load balancer. The node tells clients the address it knows itself by unless you tell it another.

Fix. Make the node announce an address clients can use. For clients behind a NAT, set cluster-announce-client-ipv4 and cluster-announce-client-port. To give clients a host name, set cluster-announce-hostname and cluster-preferred-endpoint-type hostname. See Announce addresses and NAT, which shows what MOVED, CLUSTER SLOTS and CLUSTER NODES print for each. These settings change at runtime with CONFIG SET.

The same symptom appears after you move nodes to new addresses and the other nodes still know one of them by its old address (CLUSTER NODES lists it at the old address, as disconnected). The fix is in Move the cluster to new addresses.

Confirm. MOVED names an address the client can connect to, and redis-cli -c completes the command.

MOVED and ASK shown to your client

127.0.0.1:7000> GET greeting
(error) MOVED 12714 127.0.0.1:7002

MOVED is not a failure. It tells the client which node owns the slot. A client that is not cluster aware shows it as an error: hotkv-cli and redis-cli without -c do. During a reshard you may see ASK instead:

127.0.0.1:7202> GET {t}3
(error) ASK 15891 127.0.0.1:7200

Fix. Use redis-cli -c, valkey-cli -c or a client library with cluster support (see Clients). Such a client follows MOVED and refreshes its slot map, and follows ASK by sending ASKING first.

CROSSSLOT

127.0.0.1:7001> MSET user:1 Ada user:2 Grace
(error) CROSSSLOT Keys in request don't hash to the same slot

A command, transaction or script named keys in different slots. Two keys usually are in different slots, even when the same node owns both.

Fix. Give the keys a shared hash tag so that they hash together:

127.0.0.1:7001> MSET {user:1}:name Ada {user:1}:city Oslo
OK

Confirm. CLUSTER KEYSLOT returns the same slot for each key. See Keys and commands.

TRYAGAIN

127.0.0.1:7202> MGET {t}1 {t}2
(error) TRYAGAIN Multiple keys request during rehashing of slot

The slot is being migrated, and the keys of this command are split: some already moved to the target, some still here. A multi-key command cannot be answered by either node alone.

Fix. Retry after a moment. The error clears when the slot has finished moving. A client library retries it for you. If it never clears, the migration is stuck, see A stuck migration.

LOADING on a replica

127.0.0.1:7461> DBSIZE
(error) LOADING HotKV is loading the dataset in memory

A replica answers commands that read or write data with LOADING while it replaces its data with a full copy from its primary. PING, INFO, AUTH and the like still answer. INFO replication on the replica shows the copy under way:

master_link_status:down
master_sync_in_progress:1

Fix. Wait, or read from the primary. A large data set takes a while to copy. The error ends when master_link_status is up. If you keep seeing it, the link keeps dropping and restarting the copy: look at the replica’s log and at the buffer limits in Cluster configuration.

A node does not start

The node writes why to its log and exits with status 1. These are the messages, with the output of each case:

The cluster bus port is in use.

Failed listening on port 17410 (the cluster bus): Address already in use (127.0.0.1). The node does not start.

Another process, or another node, holds the bus port (client port + 10000, or cluster-port). Remember that ports 7000 and 17000 belong together: a node on port 7000 and another on port 17000 collide, and the one that starts second fails, with the message of the port it cannot take (this one for the bus, the next one for a client port). Pick other ports.

The client port is in use.

Failed to bind to 127.0.0.1:7410 - Address already in use

nodes.conf is damaged or unreadable. A node never starts with a new identity beside a damaged file. It stops instead, so that you do not create a second node by accident:

Cluster: corrupt line in /var/lib/hotkv/7412/nodes.conf: "this is not a node table"
Cluster: /var/lib/hotkv/7412/nodes.conf exists but is unreadable, corrupt or has no line for this node (flag myself); refusing to start with a new node id. Restore the file, or remove it to start as a new node

The same message appears when the file cannot be read, with Cluster: cannot read <file> before it, usually because of its owner or mode. Restore the file from a backup. If you remove it, the node starts as a new, empty node, and the cluster still knows the old one.

A directive HotKV does not know. A typo such as cluster-enable yes would otherwise start a standalone node, so the start is refused:

*** FATAL CONFIG FILE ERROR *** Reading the configuration file, at e.conf, line 2: 'cluster-enable yes': Bad directive or wrong number of arguments
Not starting: fix the line above, or set ignore-unknown-directives yes to start with it skipped

replicaof on a cluster node. In a cluster, replicas are created with CLUSTER REPLICATE:

*** FATAL CONFIG FILE ERROR *** replicaof directive not allowed in cluster mode

cluster-bus-port-protected-mode yes without tls-cluster yes. The node refuses to start with an unauthenticated bus:

*** FATAL CONFIG FILE ERROR ***
cluster-bus-port-protected-mode is enabled but tls-cluster is disabled: the cluster bus port of this node would take any host that reaches it as a node of the cluster. Set 'tls-cluster yes', so that the other nodes must show a certificate tls-ca-cert-file trusts, or set 'cluster-bus-port-protected-mode no' if only trusted hosts can reach the port (a firewall, say)

A certificate problem stops a TLS node with an error about the certificate or key file. See Cluster over TLS.

Messages that are not errors. A node with the bus unauthenticated logs this at every start, as a reminder to keep the bus port private (or use tls-cluster):

WARNING: the cluster bus port is not authenticated: tls-cluster is off and cluster-bus-port-protected-mode is no, so any host that reaches the port can join this cluster and speak its protocol. Let only trusted hosts reach it, or set tls-cluster yes.

A new node that is not part of a cluster yet also logs Cluster: the cluster is down: this node is in a minority partition, reaching no majority of the primaries that serve slots at its first start. That clears once you create the cluster.

Nodes disagree

redis-cli --cluster check compares every node’s view of the slots. After CLUSTER DELSLOTS 100 on one node only, it reports:

redis-cli --cluster check 127.0.0.1:7420 | tail -6

   replicates b7c78d92a73b5a3022158710f04705c18c987881
[ERR] Nodes don't agree about configuration!
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

redis-cli --cluster fix repairs what it can. For an uncovered slot with no keys it asks before assigning the slot to a node. Its question is not answered by --cluster-yes, so type yes (the example pipes it in):

echo yes | redis-cli --cluster fix 127.0.0.1:7420 | tail -12

M: 11d2619876d1afbdde19becf708c47d42cd06b7a 127.0.0.1:7421
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[ERR] Nodes don't agree about configuration!
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

>>> Fixing slots coverage...
The following uncovered slots have no keys across the cluster:
[100]
Fix these slots by covering with a random node? (type 'yes' to accept): >>> Covering slot 100 with 127.0.0.1:7421

redis-cli --cluster check 127.0.0.1:7420 | tail -4

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Slots that hold keys are handled with more questions, so read what fix proposes before you accept it. See Scaling for what a reshard leaves behind.

Failover did not happen

A primary stopped, its replica is up, and nothing changed. Look at CLUSTER NODES on a healthy node and at the replica’s log.

• The replica has nofailover. cluster-replica-no-failover yes makes a replica never start an automatic failover, and CLUSTER NODES shows the flag:

  c2ef9f6568713718f0533274f2227527131474d2 127.0.0.1:7433@17433 slave,nofailover 97ab179fc6f4c81157401e3e0aa27274f856a345 0 1780065115536 4 connected

  The replica logs nothing about it. Set cluster-replica-no-failover no on the replica, and the failover starts. CLUSTER FAILOVER still works with the flag set.

• The replica never finished copying. A replica that has not completed a first full copy holds none or only part of the shard, so it refuses to take over. It logs:

  Cluster: not attempting failover for 6afaddeea2e27a07da99cd7d964819a2ae3429a1 - this replica has not completed a sync from it (never synced, or a full sync is under way), so it holds none or only part of the shard's data

• There is no majority. A failover needs a majority of the primaries that own slots to vote. With two of three primaries down, or with only two primaries, nobody can win. The survivors log the minority message from CLUSTERDOWN. Bring a primary back.

• The primary had no replica. Nothing can take over, and its slots are uncovered. See CLUSTERDOWN.

• It is slow, not stuck. Detection and the election take about cluster-node-timeout plus a few seconds. See Failure detection.

A healthy failover logs these lines on the nodes involved, which tells you how far a stuck one got:

Cluster: node 7167f52de3fdd56e7f1c7c4f543d15086d197ffd (127.0.0.1:7300) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it
Cluster: node 7167f52de3fdd56e7f1c7c4f543d15086d197ffd (127.0.0.1:7300) flagged FAIL: a majority of the primaries serving slots report it unreachable
This is the best ranked replica and can initiate the election immediately.
Cluster: failover election for epoch 7: votes asked of the primaries
Cluster: Failover vote 1/3 (need 2)
Cluster: Failover vote 2/3 (need 2)
Cluster: this replica won the failover election: it takes over as the primary

Confirm. CLUSTER NODES shows the former replica as master with the slots, and a write to one of those slots succeeds.

Flapping failures

Cluster: node cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472) flagged PFAIL: a PING to it waited 200 ms with nothing heard from it
FAIL message received from f200f9c30850677d3c09d31efa7fcb75c1c54f32 (127.0.0.1:7473) about cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472)
Cluster: cluster_state is fail now
Cluster: voted for 2b5074d18f9ecdc766eb83fdf849f5ad3f2970df (127.0.0.1:7475) in the failover election of epoch 7
Cluster: cluster_state is ok now
Cluster: node cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472) no longer flagged FAIL: reachable again, and a replica

Nodes flag a peer, then clear the flag a moment later, and the cluster state flips between fail and ok. With a timeout this low, a pause of under a second (a busy host, a garbage-collection style stall, a network blip) makes a healthy node look dead. A log like the one above appears on a cluster with a very low cluster-node-timeout (200 ms, say) when one node pauses for a fraction of a second: its replica takes over, and the old primary comes back as a replica.

Fix. Raise cluster-node-timeout. The default of 15000 ms suits most networks. It changes at runtime (CONFIG SET cluster-node-timeout 15000), on every node.

Confirm. No new flagged PFAIL lines, and cluster_current_epoch in CLUSTER INFO stops growing.

NOREPLICAS on CLUSTER SETSLOT

$ redis-cli -p 7420 CLUSTER SETSLOT 300 NODE 5b890d48a0baa4c1ef64d292c6e080b6dfc5978d TIMEOUT 2000
NOREPLICAS Not enough good replicas to write.

CLUSTER SETSLOT changes slot ownership and waits until every replica connected to the node has acknowledged the change. A replica that is connected but stuck (stopped, overloaded, or too far behind) holds it up until the timeout, and then nothing changes. A replica that is disconnected is not waited for. Without TIMEOUT, HotKV keeps waiting while a replica is slow but making progress.

Fix. Find the stuck replica: INFO replication on the primary shows each replica’s offset and lag. Restart or fix it, or wait for it to catch up, then repeat the command. Resharding tools retry this for you.

Confirm. The command answers OK.

A stuck migration

After CLUSTER SETSLOT 200 MIGRATING <node-id> (what a reshard does first) is left without being finished, redis-cli --cluster check lists the open slot:

redis-cli --cluster check 127.0.0.1:7420 | tail -5

[WARNING] Node 127.0.0.1:7420 has slots in migrating state 200.
[WARNING] Node 127.0.0.1:7423 has slots in migrating state 200.
[WARNING] The following slots are open: 200.
>>> Check slots coverage...
[OK] All 16384 slots covered.

(The second node listed is the replica of the first.) redis-cli --cluster fix closes the slot:

redis-cli --cluster fix 127.0.0.1:7420 | tail -6

[WARNING] The following slots are open: 200.
>>> Fixing open slot 200
Set as migrating in: 127.0.0.1:7420
>>> Case 4: Closing slot 200 on 127.0.0.1:7420
>>> Check slots coverage...
[OK] All 16384 slots covered.

The states of the atomic migrations (CLUSTER MIGRATESLOTS and CLUSTER MIGRATION) and how to cancel one are in Scaling.

Confirm. redis-cli --cluster check prints no [WARNING] lines about open slots.

A removed node cannot rejoin

A node that still carries its old identity (its nodes.conf, its slots or keys) is refused when you add it to a cluster, with the not empty error. A node whose nodes.conf was lost starts as a new node with its old data loaded, and cannot become a replica until the data is gone:

$ redis-cli -p 7600 CLUSTER REPLICATE ccf7564936f2071589c772ed60cc7242139f5209
ERR To set a master the node must be empty and without assigned slots.

Fix. Empty the node with FLUSHALL, and if it carries an old cluster view run CLUSTER RESET HARD. Then add it again: CLUSTER MEET, then CLUSTER REPLICATE. Backup and restore shows the full procedure.

TLS handshake failures

With tls-cluster yes, a link that fails its certificate check is logged on both ends. These are the lines for a certificate from the wrong CA, for an expired one, and for a node that does not use TLS. See Cluster over TLS for how to read them and fix each case.

Where the logs are and what to look for

Nodes log to standard output unless logfile is set. Under systemd, read it with journalctl -u hotkv-node@7000 (or -u hotkv for the packaged service). Lines worth searching for:

Line (shortened)	Meaning
flagged PFAIL: a PING to it waited N ms	This node stopped hearing from a peer for the node timeout
flagged FAIL: a majority of the primaries serving slots report it unreachable	The cluster agrees the node is down
no longer flagged FAIL: reachable again	A node came back
cluster_state is fail now / cluster_state is ok now	The state changed. A the cluster is down: line follows, with the reason
failover election for epoch N, Failover vote, won the failover election	A replica’s election
not attempting failover for ... -	A replica refuses to take over, with the reason
voted for ... in the failover election	This primary granted a vote
Replication: refused replica / refused this node's replication password	A password mismatch between a primary and a replica
TLS handshake failed, Error accepting cluster node connection, the TLS bus link to node	TLS problems on clients and on the bus
WARNING: the cluster bus port is not authenticated	The bus is plain and open to anyone who reaches it

See also

• Deploy on VMs and cloud
• Cluster over TLS
• Replicas and failover
• Scaling
• Monitoring