# Cluster troubleshooting

> Find a HotKV cluster problem by its symptom, with the exact error or log line, the cause, the fix and how to confirm it.

URL: https://docs.hotkv.com/clustering/troubleshooting/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

Start with the symptom you see. Each section ends with a way to confirm that
the fix worked. Log lines are shown without the date, time and level that
start each line in the log. Two habits save time: run
`redis-cli --cluster check` against any node first, and read the log of the
node that reports the problem (see
[Where the logs are](#where-the-logs-are-and-what-to-look-for)).

| Symptom | Likely cause | Section |
| --- | --- | --- |
| `redis-cli --cluster create` prints dots after `Waiting for the cluster to join` and never ends | The bus ports between the nodes are blocked | [Create waits forever](#create-waits-forever) |
| `[ERR] Node ... is not empty` | The node already knows other nodes, owns slots or holds keys | [A node is not empty](#a-node-is-not-empty) |
| `DENIED HotKV is running in protected mode ...` | No password, and the client is not on the same host | [DENIED](#denied-protected-mode) |
| `NOAUTH` or `WRONGPASS` from `redis-cli`, or `AUTH failed` after a redirect | Wrong or missing password, or a user that exists on some nodes only | [Authentication errors](#authentication-errors) |
| A replica shows `master_link_status:down` | The replica and primary passwords differ | [A replica cannot sync](#a-replica-cannot-sync) |
| `CLUSTERDOWN` | A slot is uncovered, or the node is on the minority side | [CLUSTERDOWN](#clusterdown) |
| `MOVED` to an address the client cannot reach | The node announces an address clients cannot use | [Redirected to an unreachable address](#redirected-to-an-unreachable-address) |
| `MOVED` shown by a client that does not follow it | The client is not cluster aware | [MOVED and ASK shown to your client](#moved-and-ask-shown-to-your-client) |
| `CROSSSLOT` | A command names keys in different slots | [CROSSSLOT](#crossslot) |
| `TRYAGAIN` | A multi-key command hit a slot that is moving | [TRYAGAIN](#tryagain) |
| `LOADING HotKV is loading the dataset in memory` on a replica | The replica is copying its primary's data | [LOADING on a replica](#loading-on-a-replica) |
| A node exits at start | Port in use, a damaged `nodes.conf`, a bad setting | [A node does not start](#a-node-does-not-start) |
| `redis-cli --cluster check` reports `[ERR]` or `[WARNING]` | Nodes disagree, slots are uncovered or open | [Nodes disagree](#nodes-disagree) |
| A primary died and no replica took over | `nofailover`, no majority, or a replica that never synced | [Failover did not happen](#failover-did-not-happen) |
| Nodes keep flagging each other as failing | `cluster-node-timeout` is too low for the network | [Flapping failures](#flapping-failures) |
| `NOREPLICAS` on `CLUSTER SETSLOT` | A replica is connected but not acknowledging | [NOREPLICAS](#noreplicas-on-cluster-setslot) |
| A slot migration seems stuck | A slot is left open on a node | [A stuck migration](#a-stuck-migration) |
| TLS handshake errors in the log | Wrong CA, expired certificate, plain node in a TLS cluster | [TLS handshake failures](#tls-handshake-failures) |

## Create waits forever

```text
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.......................................
```

`redis-cli --cluster create` introduced the nodes to each other with
`CLUSTER MEET` and now waits until they all see each other. That happens over
the cluster bus, not over the client port, so if even one pair of nodes
cannot open a connection to each other's bus port (client port + 10000), the
dots never stop. The client ports are fine, which is why the earlier steps
work. With the bus ports blocked by a firewall rule (see
[Ports and firewalls](https://docs.hotkv.com/clustering/production/index.md#ports-and-firewalls)) the command
never ends; `timeout 40` makes it end:

```bash
export REDISCLI_AUTH=change-me; timeout 40 redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes; echo "exit status $?"
```

```text
...
Waiting for the cluster to join
.......................................exit status 124
```

Check from one host whether another host's bus port answers. A blocked port
times out, and a reachable one answers at once (a plain client does not speak
the bus protocol, so the answer is a reset):

```bash
redis-cli -h 10.0.0.2 -p 17000 -t 3 PING
```

```text
Could not connect to Redis at 10.0.0.2:17000: Connection timed out
```

With the firewall rule removed, the same command prints:

```text
Error: Connection reset by peer
```

On the first node, `CLUSTER NODES` shows only the nodes on its own side, and
`CLUSTER INFO` reports `cluster_state:fail` because slots are missing:

```text
$ REDISCLI_AUTH=change-me redis-cli -p 7000 CLUSTER NODES
aca663cab2beed27d99f8c1efecf2f07fe43805d 10.0.0.1:7002@17002 master - 0 1780066857039 3 connected
28efd03d876eef35d833762acdb3f5ec7a1df046 10.0.0.1:7001@17001 master - 0 1780066856038 2 connected 10923-16383
ce65054c3ecabefa6336daecec2a1b5ab5947f9d 10.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460
$ REDISCLI_AUTH=change-me redis-cli -p 7000 CLUSTER INFO | head -n 6
cluster_enabled:1
cluster_state:fail
cluster_slots_assigned:10922
cluster_slots_ok:10922
cluster_slots_pfail:0
cluster_slots_fail:0
```

**Fix.** Open the bus ports between all nodes, both ways. See the
[firewall rules](https://docs.hotkv.com/clustering/production/index.md#ports-and-firewalls). A cluster whose
nodes announce an address the others cannot reach behaves the same, see
[Redirected to an unreachable address](#redirected-to-an-unreachable-address).

Opening the ports does not complete the interrupted create. The nodes were
told about each other once and gave up, so waiting does not help
(`CLUSTER NODES` keeps showing only the nodes on its own side), and creating again fails with
`[ERR] Node ... is not empty`. Reset the nodes and create again, as shown in
the next section.

**Confirm.** The create finishes with `[OK] All 16384 slots covered.` and
`redis-cli --cluster check` reports no errors.

## A node is not empty

```text
[ERR] Node 10.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
```

`redis-cli --cluster create` only accepts fresh nodes: no known peers, no
slots and no keys. The usual causes are an earlier failed create, a node
reused from another cluster, or a node whose old `nodes.conf` was kept.

**Fix.** If you do not need the data on these nodes, empty and reset every
node, then create again. `CLUSTER RESET HARD` gives the node a new ID and
forgets the others. A primary that holds keys refuses to reset, so run
`FLUSHALL` first:

```bash
export REDISCLI_AUTH=change-me
for node in 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002; do
  redis-cli -h ${node%:*} -p ${node#*:} CLUSTER RESET HARD
done
```

```text
OK
OK
OK
OK
OK
OK
```

To keep the data of a node, add it to a running cluster as a new node
instead of creating from it. See [Scaling](https://docs.hotkv.com/clustering/scaling/index.md).

**Confirm.** `redis-cli --cluster create` runs through.

## DENIED, protected mode

```bash
redis-cli -h 10.0.0.2 -p 7000 PING
```

```text
DENIED HotKV is running in protected mode because protected mode is on and the default user has no password: only clients on the loopback interface are served. To serve other hosts, set a password for the default user (requirepass), or turn protected mode off where the network already keeps strangers out: 'CONFIG SET protected-mode no' from the loopback interface (then 'CONFIG REWRITE' to keep it), 'protected-mode no' in the configuration file, or '--protected-mode no' on the command line.
```

The node listens on a non-loopback address, its default user has no
password, and protected mode is on (the default), so it serves only clients on
the same host. The same command run on the node's own host answers `PONG`.
At startup the node logs a warning, and it logs each client it refuses:

```text
Protected mode is on and the default user has no password, so clients from other hosts are refused (DENIED). Set a password (requirepass), or protected-mode no where the network keeps strangers out.
Refused a client from 10.0.0.1:58824: protected mode is on and the default user has no password, so only clients on a loopback address are served
```

Because nodes also reach each other's client port for replication and
resharding, this blocks replicas and `redis-cli --cluster` as well.

**Fix.** Set the same `requirepass` on every node, the right fix on any
network you do not fully control. Where the network already keeps strangers
out, you can turn protected mode off instead. See
[Authentication and protected mode](https://docs.hotkv.com/clustering/production/index.md#authentication-and-protected-mode).

**Confirm.** The same command from the other host answers
`NOAUTH Authentication required.` once a password is set, and `PONG` when
you give the password (or after you turn protected mode off).

## Authentication errors

With a password set, `redis-cli --cluster` needs it for every node, and a
missing or wrong password stops it at the first node:

```bash
redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes
```

```text
[ERR] Node 10.0.0.1:7000 NOAUTH Authentication required.
```

```bash
REDISCLI_AUTH=wrong redis-cli --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002   10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes
```

```text
Node 10.0.0.1:7000 replied with error:
WRONGPASS invalid username-password pair or user is disabled.
```

**Fix.** Give the password through the `REDISCLI_AUTH` environment variable
(or `-a`, which prints a warning that the password is visible in the process
list). Every node must accept it, so use the same `requirepass` everywhere.

**A user that exists on some nodes only.** ACL users are per node. A client
logged in as an application user on one node is redirected to a node that
does not know the user, and `redis-cli -c` stops:

```bash
REDISCLI_AUTH=app-secret redis-cli -c --user app -h 10.0.0.1 -p 7000 SET greeting hello
```

```text
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
Error:
```

Keys that live on the node where the user exists work, and the others fail,
which makes the problem look random. **Fix.** Create the same user on every
node, primaries and replicas (see
[Authentication and protected mode](https://docs.hotkv.com/clustering/production/index.md#authentication-and-protected-mode)).

**Confirm.** A write to a key on each node succeeds with the application user.

## A replica cannot sync

A replica whose password differs from its primary's never finishes its first
copy. On the replica:

```text
$ redis-cli -p 7451 INFO replication | grep -E 'role|master_link_status|master_port'
role:slave
master_port:7450
master_link_status:down
```

The replica's log says what its primary refused, and the primary's log shows
the other end:

```text
Replication: cannot connect to primary 127.0.0.1:7450: it refused this node's replication password (attempt 1, retrying)
Replication: refused replica 127.0.0.1:53216: it does not have this node's replication password (replication.password, else requirepass)
```

**Cause.** Replicas prove they know the primary's replication password. That
is `replication.password` if set, else `masterauth`, else `requirepass`.
**Fix.** Use the same `requirepass` on every node, or set `masterauth` on the
replicas to the primary's password. Both work; the change takes effect
without a restart.

**Confirm.** `master_link_status:up` on the replica.

## CLUSTERDOWN

`CLUSTERDOWN` has three forms. Which one you see tells you the cause.

**`CLUSTERDOWN Hash slot not served`.** No node owns the slot of the key: the
slot is unassigned. That happens after `CLUSTER DELSLOTS`, after `CLUSTER
FORGET` of the node that owned the slot, or on a cluster whose slots were never
assigned. A shard that is only down still owns its slots, so its keys do not
get this error: they get `The cluster is down`, or a redirect to the dead node
when `cluster-require-full-coverage` is `no`, as described below.

```text
127.0.0.1:7200> GET greeting
(error) CLUSTERDOWN Hash slot not served
```

(Slot 12714, the slot of `greeting`, had been removed with `CLUSTER DELSLOTS 12714` on
every node. A key in a slot that still has an owner gets `The cluster is down`
instead, described next, while the cluster state is `fail`.)

Find the uncovered slots with `redis-cli --cluster check`, and see
[Nodes disagree](#nodes-disagree) to fix them.

**`CLUSTERDOWN The cluster is down`.** The node's cluster state is `fail`,
and it refuses every command that names a key. There are two reasons, and the
log says which:

```text
Cluster: the cluster is down: At least one hash slot is not served by any available node, and cluster-require-full-coverage is yes
Cluster: the cluster is down: this node is in a minority partition, reaching no majority of the primaries that serve slots
```

- *A slot is uncovered* and `cluster-require-full-coverage` is `yes` (the
  default). One lost shard stops the whole cluster. If you prefer to keep
  serving the slots that are covered, set `cluster-require-full-coverage no`
  on every node. Keys in the lost shard's slots are then redirected to the
  failed node, and a client sees `Could not connect to Redis at 127.0.0.1:7430:
  Connection refused`.
- *This node is on the minority side.* It cannot reach a majority of the
  primaries that own slots, for example after a network split, or after two
  of three primaries stopped:

```text
127.0.0.1:7200> GET greeting
(error) CLUSTERDOWN The cluster is down
127.0.0.1:7200> SET greeting hi
(error) CLUSTERDOWN The cluster is down
```

  Restore the missing nodes or the network. The state returns to `ok` on its
  own when the node can see a majority again. A primary that has just been
  restarted also stays in `fail` for a few seconds before it accepts writes.

**`CLUSTERDOWN The cluster is down and only accepts read commands`.** With
`cluster-allow-reads-when-down yes`, a node in the `fail` state still serves
reads and answers writes with this error.

**Confirm.** `CLUSTER INFO` reports `cluster_state:ok` and a write to a key in
the affected slot succeeds.

## Redirected to an unreachable address

A client that follows a redirect stalls, then fails:

```text
$ redis-cli -c -t 3 -p 7420 SET greeting hello
Could not connect to Redis at 172.31.0.5:7422: Connection timed out
$ redis-cli -p 7420 GET greeting
MOVED 12714 172.31.0.5:7422
```

The first command works against the node you contacted, but the redirect
names an address that the client cannot reach from where it runs: the
node's address inside a container network, a private address behind a NAT or
a load balancer. The node tells clients the address it knows itself by unless
you tell it another.

**Fix.** Make the node announce an address clients can use. For clients behind
a NAT, set `cluster-announce-client-ipv4` and `cluster-announce-client-port`.
To give clients a host name, set `cluster-announce-hostname` and
`cluster-preferred-endpoint-type hostname`. See
[Announce addresses and NAT](https://docs.hotkv.com/clustering/production/index.md#announce-addresses-and-nat),
which shows what `MOVED`, `CLUSTER SLOTS` and `CLUSTER NODES` print for each.
These settings change at runtime with `CONFIG SET`.

The same symptom appears after you move nodes to new addresses and the other
nodes still know one of them by its old address (`CLUSTER NODES` lists it at the
old address, as `disconnected`). The fix is in
[Move the cluster to new addresses](https://docs.hotkv.com/clustering/backup-and-restore/index.md#move-the-cluster-to-new-addresses).

**Confirm.** `MOVED` names an address the client can connect to, and
`redis-cli -c` completes the command.

## MOVED and ASK shown to your client

```text
127.0.0.1:7000> GET greeting
(error) MOVED 12714 127.0.0.1:7002
```

`MOVED` is not a failure. It tells the client which node owns the slot. A
client that is not cluster aware shows it as an error: `hotkv-cli` and
`redis-cli` without `-c` do. During a reshard you may see `ASK` instead:

```text
127.0.0.1:7202> GET {t}3
(error) ASK 15891 127.0.0.1:7200
```

**Fix.** Use `redis-cli -c`, `valkey-cli -c` or a client library with cluster
support (see [Clients](https://docs.hotkv.com/clustering/clients/index.md)). Such a client follows `MOVED`
and refreshes its slot map, and follows `ASK` by sending `ASKING` first.

## CROSSSLOT

```text
127.0.0.1:7001> MSET user:1 Ada user:2 Grace
(error) CROSSSLOT Keys in request don't hash to the same slot
```

A command, transaction or script named keys in different slots. Two keys
usually are in different slots, even when the same node owns both.

**Fix.** Give the keys a shared hash tag so that they hash together:

```text
127.0.0.1:7001> MSET {user:1}:name Ada {user:1}:city Oslo
OK
```

**Confirm.** `CLUSTER KEYSLOT` returns the same slot for each key. See
[Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md).

## TRYAGAIN

```text
127.0.0.1:7202> MGET {t}1 {t}2
(error) TRYAGAIN Multiple keys request during rehashing of slot
```

The slot is being migrated, and the keys of this command are split: some
already moved to the target, some still here. A multi-key command cannot be
answered by either node alone.

**Fix.** Retry after a moment. The error clears when the slot has finished
moving. A client library retries it for you. If it never clears, the
migration is stuck, see [A stuck migration](#a-stuck-migration).

## LOADING on a replica

```text
127.0.0.1:7461> DBSIZE
(error) LOADING HotKV is loading the dataset in memory
```

A replica answers commands that read or write data with `LOADING` while it
replaces its data with a full copy from its primary. `PING`, `INFO`, `AUTH`
and the like still answer. `INFO replication` on the replica shows the copy
under way:

```text
master_link_status:down
master_sync_in_progress:1
```

**Fix.** Wait, or read from the primary. A large data set takes a while to
copy. The error ends when `master_link_status` is `up`. If you keep seeing
it, the link keeps dropping and restarting the copy: look at the replica's log
and at the buffer limits in [Cluster configuration](https://docs.hotkv.com/clustering/configuration/index.md).

## A node does not start

The node writes why to its log and exits with status 1. These are the
messages, with the output of each case:

**The cluster bus port is in use.**

```text
Failed listening on port 17410 (the cluster bus): Address already in use (127.0.0.1). The node does not start.
```

Another process, or another node, holds the bus port (client port + 10000, or
`cluster-port`). Remember that ports 7000 and 17000 belong together: a node
on port 7000 and another on port 17000 collide, and the one that starts second fails,
with the message of the port it cannot take (this one for the bus, the next one for
a client port). Pick other ports.

**The client port is in use.**

```text
Failed to bind to 127.0.0.1:7410 - Address already in use
```

**`nodes.conf` is damaged or unreadable.** A node never starts with a new
identity beside a damaged file. It stops instead, so that you do not create a
second node by accident:

```text
Cluster: corrupt line in /var/lib/hotkv/7412/nodes.conf: "this is not a node table"
Cluster: /var/lib/hotkv/7412/nodes.conf exists but is unreadable, corrupt or has no line for this node (flag myself); refusing to start with a new node id. Restore the file, or remove it to start as a new node
```

The same message appears when the file cannot be read, with
`Cluster: cannot read <file>` before it, usually because of its owner or
mode. Restore the file from a [backup](https://docs.hotkv.com/clustering/backup-and-restore/index.md). If
you remove it, the node starts as a new, empty node, and the cluster still
knows the old one.

**A directive HotKV does not know.** A typo such as `cluster-enable yes`
would otherwise start a standalone node, so the start is refused:

```text
*** FATAL CONFIG FILE ERROR *** Reading the configuration file, at e.conf, line 2: 'cluster-enable yes': Bad directive or wrong number of arguments
Not starting: fix the line above, or set ignore-unknown-directives yes to start with it skipped
```

**`replicaof` on a cluster node.** In a cluster, replicas are created with
`CLUSTER REPLICATE`:

```text
*** FATAL CONFIG FILE ERROR *** replicaof directive not allowed in cluster mode
```

**`cluster-bus-port-protected-mode yes` without `tls-cluster yes`.** The node
refuses to start with an unauthenticated bus:

```text
*** FATAL CONFIG FILE ERROR ***
cluster-bus-port-protected-mode is enabled but tls-cluster is disabled: the cluster bus port of this node would take any host that reaches it as a node of the cluster. Set 'tls-cluster yes', so that the other nodes must show a certificate tls-ca-cert-file trusts, or set 'cluster-bus-port-protected-mode no' if only trusted hosts can reach the port (a firewall, say)
```

**A certificate problem** stops a TLS node with an error about the
certificate or key file. See [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md).

**Messages that are not errors.** A node with the bus unauthenticated logs
this at every start, as a reminder to keep the bus port private (or use
`tls-cluster`):

```text
WARNING: the cluster bus port is not authenticated: tls-cluster is off and cluster-bus-port-protected-mode is no, so any host that reaches the port can join this cluster and speak its protocol. Let only trusted hosts reach it, or set tls-cluster yes.
```

A new node that is not part of a cluster yet also logs `Cluster: the cluster is
down: this node is in a minority partition, reaching no majority of the
primaries that serve slots` at its first start. That clears once you create
the cluster.

## Nodes disagree

`redis-cli --cluster check` compares every node's view of the slots. After
`CLUSTER DELSLOTS 100` on one node only, it reports:

```bash
redis-cli --cluster check 127.0.0.1:7420 | tail -6
```

```text
   replicates b7c78d92a73b5a3022158710f04705c18c987881
[ERR] Nodes don't agree about configuration!
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
```

`redis-cli --cluster fix` repairs what it can. For an uncovered slot with no
keys it asks before assigning the slot to a node. Its question is not
answered by `--cluster-yes`, so type `yes` (the example pipes it in):

```bash
echo yes | redis-cli --cluster fix 127.0.0.1:7420 | tail -12
```

```text
M: 11d2619876d1afbdde19becf708c47d42cd06b7a 127.0.0.1:7421
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[ERR] Nodes don't agree about configuration!
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

>>> Fixing slots coverage...
The following uncovered slots have no keys across the cluster:
[100]
Fix these slots by covering with a random node? (type 'yes' to accept): >>> Covering slot 100 with 127.0.0.1:7421
```

```bash
redis-cli --cluster check 127.0.0.1:7420 | tail -4
```

```text
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

Slots that hold keys are handled with more questions, so read what `fix`
proposes before you accept it. See [Scaling](https://docs.hotkv.com/clustering/scaling/index.md) for what a
reshard leaves behind.

## Failover did not happen

A primary stopped, its replica is up, and nothing changed. Look at
`CLUSTER NODES` on a healthy node and at the replica's log.

- **The replica has `nofailover`.** `cluster-replica-no-failover yes` makes a
  replica never start an automatic failover, and `CLUSTER NODES` shows the
  flag:

  ```text
  c2ef9f6568713718f0533274f2227527131474d2 127.0.0.1:7433@17433 slave,nofailover 97ab179fc6f4c81157401e3e0aa27274f856a345 0 1780065115536 4 connected
  ```

  The replica logs nothing about it. Set `cluster-replica-no-failover no` on
  the replica, and the failover starts. `CLUSTER FAILOVER` still works with
  the flag set.
- **The replica never finished copying.** A replica that has not completed a
  first full copy holds none or only part of the shard, so it refuses to take
  over. It logs:

  ```text
  Cluster: not attempting failover for 6afaddeea2e27a07da99cd7d964819a2ae3429a1 - this replica has not completed a sync from it (never synced, or a full sync is under way), so it holds none or only part of the shard's data
  ```

- **There is no majority.** A failover needs a majority of the primaries that
  own slots to vote. With two of three primaries down, or with only two
  primaries, nobody can win. The survivors log the minority message from
  [CLUSTERDOWN](#clusterdown). Bring a primary back.
- **The primary had no replica.** Nothing can take over, and its slots are
  uncovered. See [CLUSTERDOWN](#clusterdown).
- **It is slow, not stuck.** Detection and the election take about
  `cluster-node-timeout` plus a few seconds. See
  [Failure detection](https://docs.hotkv.com/clustering/production/index.md#failure-detection).

A healthy failover logs these lines on the nodes involved, which tells you
how far a stuck one got:

```text
Cluster: node 7167f52de3fdd56e7f1c7c4f543d15086d197ffd (127.0.0.1:7300) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it
Cluster: node 7167f52de3fdd56e7f1c7c4f543d15086d197ffd (127.0.0.1:7300) flagged FAIL: a majority of the primaries serving slots report it unreachable
This is the best ranked replica and can initiate the election immediately.
Cluster: failover election for epoch 7: votes asked of the primaries
Cluster: Failover vote 1/3 (need 2)
Cluster: Failover vote 2/3 (need 2)
Cluster: this replica won the failover election: it takes over as the primary
```

**Confirm.** `CLUSTER NODES` shows the former replica as `master` with the
slots, and a write to one of those slots succeeds.

## Flapping failures

```text
Cluster: node cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472) flagged PFAIL: a PING to it waited 200 ms with nothing heard from it
FAIL message received from f200f9c30850677d3c09d31efa7fcb75c1c54f32 (127.0.0.1:7473) about cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472)
Cluster: cluster_state is fail now
Cluster: voted for 2b5074d18f9ecdc766eb83fdf849f5ad3f2970df (127.0.0.1:7475) in the failover election of epoch 7
Cluster: cluster_state is ok now
Cluster: node cbe96cb8c6830fd272053c77f96379535c26c40c (127.0.0.1:7472) no longer flagged FAIL: reachable again, and a replica
```

Nodes flag a peer, then clear the flag a moment later, and the cluster state
flips between `fail` and `ok`. With a timeout this low, a pause of under a
second (a busy host, a garbage-collection style stall, a network blip) makes
a healthy node look dead. A log like the one above appears on a cluster with
a very low `cluster-node-timeout` (200 ms, say) when one node pauses for a
fraction of a second: its replica takes over, and the old primary comes back as
a replica.

**Fix.** Raise `cluster-node-timeout`. The default of 15000 ms suits most
networks. It changes at runtime (`CONFIG SET cluster-node-timeout 15000`), on
every node.

**Confirm.** No new `flagged PFAIL` lines, and `cluster_current_epoch` in
`CLUSTER INFO` stops growing.

## NOREPLICAS on CLUSTER SETSLOT

```text
$ redis-cli -p 7420 CLUSTER SETSLOT 300 NODE 5b890d48a0baa4c1ef64d292c6e080b6dfc5978d TIMEOUT 2000
NOREPLICAS Not enough good replicas to write.
```

`CLUSTER SETSLOT` changes slot ownership and waits until every replica
connected to the node has acknowledged the change. A replica that is
connected but stuck (stopped, overloaded, or too far behind) holds it up
until the timeout, and then nothing changes. A replica that is disconnected
is not waited for. Without `TIMEOUT`, HotKV keeps waiting while a replica is
slow but making progress.

**Fix.** Find the stuck replica: `INFO replication` on the primary shows each
replica's offset and lag. Restart or fix it, or wait for it to catch up, then
repeat the command. Resharding tools retry this for you.

**Confirm.** The command answers `OK`.

## A stuck migration

After `CLUSTER SETSLOT 200 MIGRATING <node-id>` (what a reshard does first) is
left without being finished, `redis-cli --cluster check` lists the open slot:

```bash
redis-cli --cluster check 127.0.0.1:7420 | tail -5
```

```text
[WARNING] Node 127.0.0.1:7420 has slots in migrating state 200.
[WARNING] Node 127.0.0.1:7423 has slots in migrating state 200.
[WARNING] The following slots are open: 200.
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

(The second node listed is the replica of the first.) `redis-cli --cluster fix`
closes the slot:

```bash
redis-cli --cluster fix 127.0.0.1:7420 | tail -6
```

```text
[WARNING] The following slots are open: 200.
>>> Fixing open slot 200
Set as migrating in: 127.0.0.1:7420
>>> Case 4: Closing slot 200 on 127.0.0.1:7420
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

The states of the atomic migrations (`CLUSTER MIGRATESLOTS` and
`CLUSTER MIGRATION`) and how to cancel one are in
[Scaling](https://docs.hotkv.com/clustering/scaling/index.md).

**Confirm.** `redis-cli --cluster check` prints no `[WARNING]` lines about
open slots.

## A removed node cannot rejoin

A node that still carries its old identity (its `nodes.conf`, its slots or
keys) is refused when you add it to a cluster, with the
[not empty](#a-node-is-not-empty) error. A node whose `nodes.conf` was lost
starts as a new node with its old data loaded, and cannot become a replica
until the data is gone:

```text
$ redis-cli -p 7600 CLUSTER REPLICATE ccf7564936f2071589c772ed60cc7242139f5209
ERR To set a master the node must be empty and without assigned slots.
```

**Fix.** Empty the node with `FLUSHALL`, and if it carries an old cluster view
run `CLUSTER RESET HARD`. Then add it again: `CLUSTER MEET`, then
`CLUSTER REPLICATE`. [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md)
shows the full procedure.

## TLS handshake failures

With `tls-cluster yes`, a link that fails its certificate check is logged on
both ends. These are the lines for a certificate from the wrong CA, for an
expired one, and for a node that does not use TLS. See
[Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md#when-the-handshake-fails) for how to read
them and fix each case.

## Where the logs are and what to look for

Nodes log to standard output unless `logfile` is set. Under systemd, read it
with `journalctl -u hotkv-node@7000` (or `-u hotkv` for the packaged
service). Lines worth searching for:

| Line (shortened) | Meaning |
| --- | --- |
| `flagged PFAIL: a PING to it waited N ms` | This node stopped hearing from a peer for the node timeout |
| `flagged FAIL: a majority of the primaries serving slots report it unreachable` | The cluster agrees the node is down |
| `no longer flagged FAIL: reachable again` | A node came back |
| `cluster_state is fail now` / `cluster_state is ok now` | The state changed. A `the cluster is down:` line follows, with the reason |
| `failover election for epoch N`, `Failover vote`, `won the failover election` | A replica's election |
| `not attempting failover for ... - ` | A replica refuses to take over, with the reason |
| `voted for ... in the failover election` | This primary granted a vote |
| `Replication: refused replica` / `refused this node's replication password` | A password mismatch between a primary and a replica |
| `TLS handshake failed`, `Error accepting cluster node connection`, `the TLS bus link to node` | TLS problems on clients and on the bus |
| `WARNING: the cluster bus port is not authenticated` | The bus is plain and open to anyone who reaches it |

## See also

- [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md)
- [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md)
- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md)
- [Scaling](https://docs.hotkv.com/clustering/scaling/index.md)
- [Monitoring](https://docs.hotkv.com/clustering/monitoring/index.md)