# Monitoring a cluster

> What to watch on a HotKV cluster, the fields and flags that mean trouble, the metrics endpoint, and a health-check script with alert conditions.

URL: https://docs.hotkv.com/clustering/monitoring/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

A cluster is healthy when every slot has a working owner, no node is flagged as
failing, every replica follows its primary, and no migration is stuck. HotKV reports
all of that through commands you can poll: `CLUSTER INFO`, `CLUSTER NODES`, `CLUSTER
SHARDS`, `CLUSTER LINKS` and `INFO`. It does not export cluster state to the Prometheus
endpoint (see [Metrics endpoint](#metrics-endpoint)), so a cluster monitor polls those
commands, and the script at the end of this page does it for you. For monitoring that
applies to any server (memory, slow log, clients), see [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md).

The examples use a cluster of three primaries and three replicas on `127.0.0.1`, ports 7000 to 7005, started with
`--cluster-node-timeout 5000` and `--appendonly yes`.

## The quick checks

Two commands tell you most of it. `redis-cli --cluster check` exits with status 0 when
the nodes agree about the slots, every slot is covered and none is open, and with 1
otherwise, so it works from cron or a monitoring agent:

```
$ redis-cli --cluster check 127.0.0.1:7000 > /dev/null; echo exit=$?
exit=0
```

`CLUSTER INFO` on any node gives the state as that node sees it:

```
127.0.0.1:7000> CLUSTER INFO
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_nodes_pfail:0
cluster_nodes_fail:0
cluster_voting_nodes_pfail:0
cluster_voting_nodes_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:17
cluster_stats_messages_pong_sent:19
cluster_stats_messages_sent:36
cluster_stats_messages_ping_received:14
cluster_stats_messages_pong_received:20
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:39
cluster_stats_bytes_sent:96684
cluster_stats_bytes_received:102336
cluster_stats_pubsub_bytes_sent:0
cluster_stats_pubsub_bytes_received:0
cluster_stats_module_bytes_sent:0
cluster_stats_module_bytes_received:0
total_cluster_links_buffer_limit_exceeded:0
cluster_slot_migration_active_tasks:0
cluster_slot_migration_active_trim_running:0
cluster_slot_migration_active_trim_current_job_keys:0
cluster_slot_migration_active_trim_current_job_trimmed:0
cluster_slot_migration_stats_active_trim_started:0
cluster_slot_migration_stats_active_trim_completed:0
cluster_slot_migration_stats_active_trim_cancelled:0
```

[`CLUSTER INFO`](https://docs.hotkv.com/commands/cluster-info/index.md) describes the cluster from the point of view of
one node, so ask more than one node when you suspect a split. The fields to watch:

| Field | Healthy | A bad value means |
| --- | --- | --- |
| `cluster_state` | `ok` | This node refuses keyed commands (`CLUSTERDOWN`): some slot has no owner while `cluster-require-full-coverage` is on, or this node cannot reach a majority of the primaries. |
| `cluster_slots_assigned` | `16384` | Slots with no owner. |
| `cluster_slots_ok` | `16384` | Slots whose owner this node considers failing or suspect. |
| `cluster_slots_pfail`, `cluster_slots_fail` | `0` | Slots owned by a node that is suspected (`pfail`) or marked failed (`fail`). |
| `cluster_nodes_pfail`, `cluster_nodes_fail` | `0` | Nodes of any role suspected or failed. A failed replica is not a slot problem, but it is one failure from one. |
| `cluster_voting_nodes_pfail`, `cluster_voting_nodes_fail` | `0` | The same, counting only the primaries that serve slots and vote in elections. |
| `cluster_known_nodes` | The number of nodes you run | Fewer: a node was forgotten or never joined. More: a node you did not expect, or one in handshake. |
| `cluster_size` | The number of primaries with slots | A shard lost all its slots, or one is missing. |
| `cluster_current_epoch` | Rises on a failover and on a slot move | It rose without a planned operation: a failover happened. Compare it across checks. |
| `total_cluster_links_buffer_limit_exceeded` | `0` | A bus link was dropped because its send buffer passed `cluster-link-sendbuf-limit`. |
| `cluster_slot_migration_active_tasks` | `0` outside a reshard | An [atomic slot migration](https://docs.hotkv.com/clustering/scaling/index.md#watch-a-migration) is running, or stuck. |

The `cluster_stats_*` fields count the messages and bytes of the cluster bus, and
`cluster_stats_pubsub_bytes_*` the bytes of pub/sub messages the bus carries.

## CLUSTER NODES

[`CLUSTER NODES`](https://docs.hotkv.com/commands/cluster-nodes/index.md) lists every node with its role, flags, the link
state and its slots:

```
127.0.0.1:7000> CLUSTER NODES
4d2f498e0c4081dd606c400ebe9e0f91e6eb6a17 127.0.0.1:7005@17005 slave 5c56a282622e115a5d9a5e8de3549977e2f20b70 0 1791065594700 6 connected
3220ad57d4e69e3c6212a70da54976e268d7fda3 127.0.0.1:7004@17004 slave 0a63a5044dafa2068fe65424b33cb17456f6948c 0 1791065594199 5 connected
1b3f10c3a2eae6522147316b7aa62c652706ae33 127.0.0.1:7003@17003 slave e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac 0 1791065593698 4 connected
5c56a282622e115a5d9a5e8de3549977e2f20b70 127.0.0.1:7002@17002 master - 0 1791065592697 3 connected 10923-16383
0a63a5044dafa2068fe65424b33cb17456f6948c 127.0.0.1:7001@17001 master - 0 1791065593899 2 connected 5461-10922
e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac 127.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460
```

The fields are the node ID, its address with the bus port after the `@`, the flags, the
primary's ID for a replica, the time of the last ping sent and pong received, the
config epoch, the link state and the slots. Look at the flags:

| Flag | Meaning |
| --- | --- |
| `myself` | The node you asked. |
| `master`, `slave` | The role. |
| `fail?` | The node did not answer for `cluster-node-timeout`: suspected (PFAIL). |
| `fail` | A majority of the primaries that serve slots agree it is down. |
| `handshake` | The node is being introduced and is not yet part of the cluster. A node that stays in handshake for long never answered. |
| `noaddr` | The node's address is not known. |
| `nofailover` | A replica set to never start an automatic failover (`cluster-replica-no-failover yes`). |
| `noflags` | No flag applies. |

The link state after the flags is `connected` or `disconnected`. Example
lines, with the columns cut down to the address, the flags, the link state and the
slots:

```
127.0.0.1:7099@17099 master,handshake disconnected
127.0.0.1:7003@17003 slave connected
127.0.0.1:7003@17003 slave,fail connected
10.0.0.1:7000@17000 master,fail? - disconnected 0-5460
10.0.0.2:7002@17002 master,fail - disconnected
127.0.0.1:7104@17104 slave,nofailover e25c9fc75c319b0b26c52a7c3f461e7ecd74bd01 connected
```

(The first is a node introduced with `CLUSTER MEET` to a port where nothing listens; it
leaves the list after about 20 seconds. The next two are the same replica before and after it
stops answering. `master,fail?` is a primary a node suspects but the majority has not yet
marked, and `master,fail` one the majority marked as failed (see
[Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md#network-partitions)).
The last line is a replica with `cluster-replica-no-failover yes`.) During a manual slot move, a slot also appears at the
end of a line as `[3311->-target-id]` on the source and `[3311-<-source-id]` on the
target (see [Scaling](https://docs.hotkv.com/clustering/scaling/index.md#watch-a-migration)). A `fail` flag on a
replica does not stop the cluster. A `fail` on a primary with no replica does.

## CLUSTER SHARDS

[`CLUSTER SHARDS`](https://docs.hotkv.com/commands/cluster-shards/index.md) groups the nodes by shard and gives each one a
`health` and its replication offset, which is the quickest way to see a lagging or
failed replica next to its primary. `health` is `online`, `fail` for a node the
cluster considers failed, or `loading` for a replica whose offset is still 0. This is
shard 1 while its replica was frozen:

```
127.0.0.1:7000> CLUSTER SHARDS
1) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac"
          3) "port"
          4) (integer) 7000
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 914085
         13) "health"
         14) "online"
      2)  1) "id"
          2) "1b3f10c3a2eae6522147316b7aa62c652706ae33"
          3) "port"
          4) (integer) 7003
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 914085
         13) "health"
         14) "fail"
   5) "id"
   6) "7e1e44b291ba4bf17d4a3d2cddb7b95a728bcd48"
...
```

The offsets are what each node last reported. They are equal when the replica is
caught up. Offsets right after startup can be 0 or 1 until the nodes have exchanged a few
messages.

## CLUSTER LINKS

Every node keeps two bus connections to every other node, one it opened (`to`) and one
the other node opened (`from`). [`CLUSTER LINKS`](https://docs.hotkv.com/commands/cluster-links/index.md) shows
them with the bytes waiting to be sent:

```
127.0.0.1:7000> CLUSTER LINKS
 1)  1) "direction"
     2) "to"
     3) "node"
     4) "36b7987885ef18024af27377756cb15d7fbc981b"
     5) "create-time"
     6) (integer) 1791066223678
     7) "events"
     8) "r"
     9) "send-buffer-allocated"
    10) (integer) 48
    11) "send-buffer-used"
    12) (integer) 0
...
```

A link whose `send-buffer-allocated` keeps growing leads to a node that does not read its bus
traffic. When it passes `cluster-link-sendbuf-limit` (default `256mb`, and `0` for no limit),
the node drops the link and opens it again, and counts it in
`total_cluster_links_buffer_limit_exceeded`. `INFO memory` reports the memory of all links as
`mem_cluster_links`.

## INFO

`INFO` carries the cluster counters next to the node's other fields. The cluster has two
sections: `INFO cluster` holds `cluster_enabled`, and `INFO cluster_info` repeats the
`CLUSTER INFO` fields, so a monitor that only reads `INFO` sees them:

```
$ redis-cli -p 7000 INFO cluster
# Cluster
cluster_enabled:1
$ redis-cli -p 7000 INFO cluster_info | head -n 6
# Cluster Info
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
```

The cluster fields in the other sections:

```
$ redis-cli -p 7000 INFO everything | grep -iE "cluster|link|slot_|migrat" | sort
# Cluster
# Cluster Info
cluster_connections:10
cluster_current_epoch:6
cluster_enabled:1
cluster_enabled:1
cluster_known_nodes:6
cluster_my_epoch:1
...
cluster_size:3
...
cluster_state:ok
...
cmdstat_cluster|info:calls=6,usec=283,usec_per_call=47.25,rejected_calls=0,failed_calls=0
...
mem_cluster_links:11680
mem_cluster_slot_export:0
mem_cluster_slot_import:0
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0
mem_cluster_slot_migration_output_buffer:0
migrate_cached_sockets:0
redis_mode:cluster
total_cluster_links_buffer_limit_exceeded:0
total_net_cluster_slot_export_bytes:0
total_net_cluster_slot_import_bytes:0
```

`cluster_connections` is the number of bus connections, and `redis_mode:cluster` tells a
client it talks to a cluster node. Replication is in `INFO replication`. On a primary it lists its
replicas with their offset and lag in seconds, and on a replica it gives the link status
and the offsets:

```
$ redis-cli -p 7000 INFO replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=7003,state=online,offset=914085,lag=0
master_failover_state:no-failover
master_replid:bd114f8adce4b5fa852f03fb2e5f26c571d8f24c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:914085
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:67108864
repl_backlog_first_byte_offset:1
repl_backlog_histlen:914085
$ redis-cli -p 7003 INFO replication
# Replication
role:slave
master_host:127.0.0.1
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:914085
slave_repl_offset:914085
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:bd114f8adce4b5fa852f03fb2e5f26c571d8f24c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:914085
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:67108864
repl_backlog_first_byte_offset:1
repl_backlog_histlen:914085
```

(The output is shortened to the fields discussed here.)

The replica is caught up when `slave_repl_offset` equals the primary's
`master_repl_offset`. `master_link_status:up` and `master_sync_in_progress:0` say the
link is established and no full copy is running. `INFO stats` on a primary counts the
syncs: `sync_full`, `sync_partial_ok` and `sync_partial_err`. A rising `sync_full` means
replicas copy everything again: they fell out of the backlog (`repl-backlog-size`) or were restarted.

### Memory a migration holds

`INFO memory` has three fields for atomic slot migrations:
`mem_cluster_slot_migration_output_buffer` is what the source holds for the
migrations it exports, `mem_cluster_slot_migration_input_buffer` what a node that
imports slots holds (the writes it keeps while the copy arrives, and input not parsed
yet), and `mem_cluster_slot_migration_input_buffer_peak` the most the second one
held. They are `0` when no migration is running, apart from the peak. With
[`replica-full-sync-buffer-limit`](https://docs.hotkv.com/clustering/scaling/index.md#writes-made-during-the-copy)
set on a target, the input buffer rises while the copy arrives. `INFO stats` counts
the bytes a node exported for slot migrations in `total_net_cluster_slot_export_bytes`.

## Slot statistics

`CLUSTER SLOT-STATS` shows the keys, memory, CPU and network bytes of each slot, which
is how you find the hot or heavy slots before you rebalance. See [Scaling a
cluster](https://docs.hotkv.com/clustering/scaling/index.md#balance-by-load).

## Metrics endpoint

HotKV serves Prometheus metrics on the port set by `metrics-port` (restart only, bound to
`127.0.0.1`; see [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md#prometheus-metrics)). A cluster node serves the
same series as any other node. **None of them is about the cluster**: nothing
for the state, the nodes, the slots, the links or the migrations:

```
$ curl -s http://127.0.0.1:9000/metrics | grep -iE "cluster|slot|migrat"; echo exit=$?
exit=1
```

The replication series are there. On a replica they show the role, the replication offset and the lag, and on
a primary the number of replicas:

```
$ curl -s http://127.0.0.1:9001/metrics | grep -E "^hotkv_replication"
hotkv_replication_role{role="replica"} 0
hotkv_replication_offset 911423
hotkv_replication_lag_microseconds 0
hotkv_replication_connected_replicas 0
$ redis-cli -p 7001 INFO replication | grep -E "^(role|master_repl_offset|slave_repl_offset|master_link_status)"
role:slave
master_link_status:up
slave_repl_offset:911423
master_repl_offset:911423
$ curl -s http://127.0.0.1:9000/metrics | grep -E "^hotkv_replication"
hotkv_replication_role{role="primary"} 1
hotkv_replication_offset 0
hotkv_replication_lag_microseconds 0
hotkv_replication_connected_replicas 1
```

So scrape the metrics port of each node for memory, clients, commands and replication, and
for the cluster itself use a script that polls `CLUSTER INFO` and `CLUSTER NODES` and
exposes the result in your monitoring system, as in the next section. A common
way is to run the script from your monitoring agent and alert when it exits with 1.

## A health-check script

This script exits with status 0 when the cluster looks healthy and 1 when it does not,
and prints one line for each problem. It reads `CLUSTER INFO`, checks every node's flags
and bus link, asks every replica whether its link to its primary is up, requires that
every primary that serves slots has at least `MIN_REPLICAS` replicas (default 1), and
runs `redis-cli --cluster check` to catch open slots and nodes that disagree about
the slots. Set `REDISCLI_AUTH` when the nodes need a password.

```bash
#!/usr/bin/env bash
# Usage: check-cluster.sh HOST:PORT
# Exits 0 when the cluster looks healthy and 1 when it does not, and prints what is wrong.
# Set REDISCLI_AUTH to the password when the nodes need one.
set -u
seed=${1:?usage: check-cluster.sh HOST:PORT}
min_replicas=${MIN_REPLICAS:-1}
rc() { timeout 5 redis-cli -h "${1%:*}" -p "${1##*:}" "${@:2}"; }
problems=0
bad() { echo "PROBLEM: $*"; problems=$((problems + 1)); }

info=$(rc "$seed" CLUSTER INFO 2>&1) || { echo "PROBLEM: cannot reach $seed: $info"; exit 1; }
field() { printf '%s\n' "$info" | tr -d '\r' | awk -F: -v k="$1" '$1 == k { print $2 }'; }

[ "$(field cluster_state)" = ok ] || bad "cluster_state is $(field cluster_state)"
[ "$(field cluster_slots_ok)" = 16384 ] || bad "only $(field cluster_slots_ok) of 16384 slots are ok"
for f in cluster_slots_pfail cluster_slots_fail cluster_nodes_pfail cluster_nodes_fail; do
  [ "$(field "$f")" = 0 ] || bad "$f is $(field "$f")"
done

declare -A replicas   # primary id -> number of replicas, for primaries that serve slots
while read -r id addr flags master _ _ _ link slots; do
  node=${addr%%@*}
  case $flags in *fail*|*noaddr*|*handshake*) bad "$node has flags $flags" ;; esac
  [ "$link" = connected ] || bad "$node: bus link is $link"
  case $flags in
    *master*) [ -n "$slots" ] && replicas[$id]=${replicas[$id]:-0} ;;
    *slave*)
      replicas[$master]=$(( ${replicas[$master]:-0} + 1 ))
      status=$(rc "$node" INFO replication 2>/dev/null | tr -d '\r' | awk -F: '$1 == "master_link_status" { print $2 }')
      [ "$status" = up ] || bad "replica $node: master_link_status is ${status:-unknown}" ;;
  esac
done < <(rc "$seed" CLUSTER NODES)

for id in "${!replicas[@]}"; do
  [ "${replicas[$id]}" -ge "$min_replicas" ] || bad "primary $id has ${replicas[$id]} replicas, want $min_replicas"
done

# open slots and slots that nodes disagree about
timeout 60 redis-cli --cluster check "$seed" > /dev/null 2>&1 || bad "redis-cli --cluster check reports a problem"

if [ "$problems" -eq 0 ]; then echo "OK: cluster_state ok, $(field cluster_known_nodes) nodes"; exit 0; fi
exit 1
```

Run healthy, it prints one line. With a replica frozen (`kill -STOP`) it names the replica:

```
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
OK: cluster_state ok, 6 nodes
exit=0
$ kill -STOP <pid>
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cluster_nodes_fail is 1
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
PROBLEM: redis-cli --cluster check reports a problem
exit=1
$ kill -CONT <pid>
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
OK: cluster_state ok, 6 nodes
exit=0
```

A slot left open by a migration, and a slot that was removed on one node, are caught by the
last check even though `cluster_state` stays `ok`:

```
127.0.0.1:7000> CLUSTER SETSLOT 100 MIGRATING 2479e739f79694cf9134902aecfb535a5bb50a54
OK
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: redis-cli --cluster check reports a problem
exit=1
127.0.0.1:7000> CLUSTER SETSLOT 100 STABLE
OK
127.0.0.1:7000> CLUSTER DELSLOTS 5
OK
$ ./check-cluster.sh 127.0.0.1:7001; echo exit=$?
PROBLEM: redis-cli --cluster check reports a problem
exit=1
127.0.0.1:7000> CLUSTER ADDSLOTS 5
OK
```

A replica that is down, and then a primary and its replica lost together, which leaves
slots without an owner:

```
$ redis-cli -p 7003 SHUTDOWN NOSAVE
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cluster_nodes_fail is 1
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: 127.0.0.1:7003: bus link is disconnected
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
exit=1
$ kill -9 <pid> <pid>
$ ./check-cluster.sh 127.0.0.1:7001; echo exit=$?
PROBLEM: cluster_state is fail
PROBLEM: only 10923 of 16384 slots are ok
PROBLEM: cluster_slots_fail is 5461
PROBLEM: cluster_nodes_fail is 2
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: 127.0.0.1:7003: bus link is disconnected
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
PROBLEM: 127.0.0.1:7000 has flags master,fail
PROBLEM: 127.0.0.1:7000: bus link is disconnected
PROBLEM: redis-cli --cluster check reports a problem
exit=1
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cannot reach 127.0.0.1:7000: Could not connect to Redis at 127.0.0.1:7000: Connection refused
exit=1
```

Point the script at a node that is not the one you suspect, and at a second node if the first
does not answer: a monitor that asks only one node is blind when that node is the one that
failed. Run it every minute from cron or from your monitoring agent, and alert on exit
status 1.

## Alert conditions

These are suggestions: thresholds are yours to set, and each is tied to a field you can
read with the commands above.

| Alert on | Field or command | Why |
| --- | --- | --- |
| The cluster is down | `cluster_state` is not `ok` on a node, for more than a few seconds | Keyed commands fail on that node. |
| Slots without a healthy owner | `cluster_slots_ok` below 16384, or `cluster_slots_fail` above 0 | A shard is unavailable. |
| A node failing | `cluster_nodes_fail` above 0, or a `fail` flag in `CLUSTER NODES`, for more than a minute | A node is down and has not come back. Page for a primary without a replica. |
| A node suspected for long | `cluster_nodes_pfail` above 0 across several checks | A node answers badly, which often precedes a failover. |
| A replica not following | `master_link_status:down` in `INFO replication` on a replica, or fewer `connected_slaves` than expected on a primary | The shard has no spare copy. |
| A replica far behind | `slave_repl_offset` well below the primary's `master_repl_offset`, or `lag` in the primary's `slave0` line | A failover now would lose that much. |
| Replicas copying everything again | `sync_full` in `INFO stats` rising | The replica fell out of the backlog or keeps restarting. |
| A failover happened | `cluster_current_epoch` rose, or the primary of a shard changed in `CLUSTER NODES` | Look at why the old primary stopped. |
| A node with the wrong number of replicas | The script's replica count below `MIN_REPLICAS` | One more failure takes a shard down. |
| An open or inconsistent slot | `redis-cli --cluster check` exits with 1 | A reshard did not finish (see [Scaling](https://docs.hotkv.com/clustering/scaling/index.md#when-a-move-fails)). |
| A stuck migration | `cluster_slot_migration_active_tasks` above 0 for longer than your largest move takes, or a `failed` state in `CLUSTER GETSLOTMIGRATIONS` | See [Scaling](https://docs.hotkv.com/clustering/scaling/index.md#watch-a-migration). |
| Bus trouble | `total_cluster_links_buffer_limit_exceeded` rising, or `send-buffer-allocated` growing in `CLUSTER LINKS` | A node does not keep up with the bus. |
| Memory | `used_memory` against `maxmemory` on every primary | A full primary answers `OOM`, and a move onto it fails. |

## Next steps

- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md): what each of these states
  looks like during a failure.
- [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md): symptoms and fixes.
- [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md): memory, slow log, latency and the other server metrics.
- [Production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md).