Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Monitoring a cluster

Last updated
Not on HotKV CloudOn-Premise: no license needed

A cluster is healthy when every slot has a working owner, no node is flagged as failing, every replica follows its primary, and no migration is stuck. HotKV reports all of that through commands you can poll: CLUSTER INFO, CLUSTER NODES, CLUSTER SHARDS, CLUSTER LINKS and INFO. It does not export cluster state to the Prometheus endpoint (see Metrics endpoint), so a cluster monitor polls those commands, and the script at the end of this page does it for you. For monitoring that applies to any server (memory, slow log, clients), see Monitoring.

The examples use a cluster of three primaries and three replicas on 127.0.0.1, ports 7000 to 7005, started with --cluster-node-timeout 5000 and --appendonly yes.

The quick checks

Section titled “The quick checks”

Two commands tell you most of it. redis-cli --cluster check exits with status 0 when the nodes agree about the slots, every slot is covered and none is open, and with 1 otherwise, so it works from cron or a monitoring agent:

$ redis-cli --cluster check 127.0.0.1:7000 > /dev/null; echo exit=$?
exit=0

CLUSTER INFO on any node gives the state as that node sees it:

127.0.0.1:7000> CLUSTER INFO
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_nodes_pfail:0
cluster_nodes_fail:0
cluster_voting_nodes_pfail:0
cluster_voting_nodes_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:17
cluster_stats_messages_pong_sent:19
cluster_stats_messages_sent:36
cluster_stats_messages_ping_received:14
cluster_stats_messages_pong_received:20
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:39
cluster_stats_bytes_sent:96684
cluster_stats_bytes_received:102336
cluster_stats_pubsub_bytes_sent:0
cluster_stats_pubsub_bytes_received:0
cluster_stats_module_bytes_sent:0
cluster_stats_module_bytes_received:0
total_cluster_links_buffer_limit_exceeded:0
cluster_slot_migration_active_tasks:0
cluster_slot_migration_active_trim_running:0
cluster_slot_migration_active_trim_current_job_keys:0
cluster_slot_migration_active_trim_current_job_trimmed:0
cluster_slot_migration_stats_active_trim_started:0
cluster_slot_migration_stats_active_trim_completed:0
cluster_slot_migration_stats_active_trim_cancelled:0

CLUSTER INFO describes the cluster from the point of view of one node, so ask more than one node when you suspect a split. The fields to watch:

Field Healthy A bad value means
cluster_state ok This node refuses keyed commands (CLUSTERDOWN): some slot has no owner while cluster-require-full-coverage is on, or this node cannot reach a majority of the primaries.
cluster_slots_assigned 16384 Slots with no owner.
cluster_slots_ok 16384 Slots whose owner this node considers failing or suspect.
cluster_slots_pfail, cluster_slots_fail 0 Slots owned by a node that is suspected (pfail) or marked failed (fail).
cluster_nodes_pfail, cluster_nodes_fail 0 Nodes of any role suspected or failed. A failed replica is not a slot problem, but it is one failure from one.
cluster_voting_nodes_pfail, cluster_voting_nodes_fail 0 The same, counting only the primaries that serve slots and vote in elections.
cluster_known_nodes The number of nodes you run Fewer: a node was forgotten or never joined. More: a node you did not expect, or one in handshake.
cluster_size The number of primaries with slots A shard lost all its slots, or one is missing.
cluster_current_epoch Rises on a failover and on a slot move It rose without a planned operation: a failover happened. Compare it across checks.
total_cluster_links_buffer_limit_exceeded 0 A bus link was dropped because its send buffer passed cluster-link-sendbuf-limit.
cluster_slot_migration_active_tasks 0 outside a reshard An atomic slot migration is running, or stuck.

The cluster_stats_* fields count the messages and bytes of the cluster bus, and cluster_stats_pubsub_bytes_* the bytes of pub/sub messages the bus carries.

CLUSTER NODES

Section titled “CLUSTER NODES”

CLUSTER NODES lists every node with its role, flags, the link state and its slots:

127.0.0.1:7000> CLUSTER NODES
4d2f498e0c4081dd606c400ebe9e0f91e6eb6a17 127.0.0.1:7005@17005 slave 5c56a282622e115a5d9a5e8de3549977e2f20b70 0 1791065594700 6 connected
3220ad57d4e69e3c6212a70da54976e268d7fda3 127.0.0.1:7004@17004 slave 0a63a5044dafa2068fe65424b33cb17456f6948c 0 1791065594199 5 connected
1b3f10c3a2eae6522147316b7aa62c652706ae33 127.0.0.1:7003@17003 slave e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac 0 1791065593698 4 connected
5c56a282622e115a5d9a5e8de3549977e2f20b70 127.0.0.1:7002@17002 master - 0 1791065592697 3 connected 10923-16383
0a63a5044dafa2068fe65424b33cb17456f6948c 127.0.0.1:7001@17001 master - 0 1791065593899 2 connected 5461-10922
e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac 127.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460

The fields are the node ID, its address with the bus port after the @, the flags, the primary’s ID for a replica, the time of the last ping sent and pong received, the config epoch, the link state and the slots. Look at the flags:

Flag Meaning
myself The node you asked.
master, slave The role.
fail? The node did not answer for cluster-node-timeout: suspected (PFAIL).
fail A majority of the primaries that serve slots agree it is down.
handshake The node is being introduced and is not yet part of the cluster. A node that stays in handshake for long never answered.
noaddr The node’s address is not known.
nofailover A replica set to never start an automatic failover (cluster-replica-no-failover yes).
noflags No flag applies.

The link state after the flags is connected or disconnected. Example lines, with the columns cut down to the address, the flags, the link state and the slots:

127.0.0.1:7099@17099 master,handshake disconnected
127.0.0.1:7003@17003 slave connected
127.0.0.1:7003@17003 slave,fail connected
10.0.0.1:7000@17000 master,fail? - disconnected 0-5460
10.0.0.2:7002@17002 master,fail - disconnected
127.0.0.1:7104@17104 slave,nofailover e25c9fc75c319b0b26c52a7c3f461e7ecd74bd01 connected

(The first is a node introduced with CLUSTER MEET to a port where nothing listens; it leaves the list after about 20 seconds. The next two are the same replica before and after it stops answering. master,fail? is a primary a node suspects but the majority has not yet marked, and master,fail one the majority marked as failed (see Replicas and failover). The last line is a replica with cluster-replica-no-failover yes.) During a manual slot move, a slot also appears at the end of a line as [3311->-target-id] on the source and [3311-<-source-id] on the target (see Scaling). A fail flag on a replica does not stop the cluster. A fail on a primary with no replica does.

CLUSTER SHARDS

Section titled “CLUSTER SHARDS”

CLUSTER SHARDS groups the nodes by shard and gives each one a health and its replication offset, which is the quickest way to see a lagging or failed replica next to its primary. health is online, fail for a node the cluster considers failed, or loading for a replica whose offset is still 0. This is shard 1 while its replica was frozen:

127.0.0.1:7000> CLUSTER SHARDS
1) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "e0b9ad81efd2c1e652ee7e6ba22dcc63e3aa03ac"
          3) "port"
          4) (integer) 7000
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 914085
         13) "health"
         14) "online"
      2)  1) "id"
          2) "1b3f10c3a2eae6522147316b7aa62c652706ae33"
          3) "port"
          4) (integer) 7003
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 914085
         13) "health"
         14) "fail"
   5) "id"
   6) "7e1e44b291ba4bf17d4a3d2cddb7b95a728bcd48"
...

The offsets are what each node last reported. They are equal when the replica is caught up. Offsets right after startup can be 0 or 1 until the nodes have exchanged a few messages.

Section titled “CLUSTER LINKS”

Every node keeps two bus connections to every other node, one it opened (to) and one the other node opened (from). CLUSTER LINKS shows them with the bytes waiting to be sent:

127.0.0.1:7000> CLUSTER LINKS
 1)  1) "direction"
     2) "to"
     3) "node"
     4) "36b7987885ef18024af27377756cb15d7fbc981b"
     5) "create-time"
     6) (integer) 1791066223678
     7) "events"
     8) "r"
     9) "send-buffer-allocated"
    10) (integer) 48
    11) "send-buffer-used"
    12) (integer) 0
...

A link whose send-buffer-allocated keeps growing leads to a node that does not read its bus traffic. When it passes cluster-link-sendbuf-limit (default 256mb, and 0 for no limit), the node drops the link and opens it again, and counts it in total_cluster_links_buffer_limit_exceeded. INFO memory reports the memory of all links as mem_cluster_links.

INFO

Section titled “INFO”

INFO carries the cluster counters next to the node’s other fields. The cluster has two sections: INFO cluster holds cluster_enabled, and INFO cluster_info repeats the CLUSTER INFO fields, so a monitor that only reads INFO sees them:

$ redis-cli -p 7000 INFO cluster
# Cluster
cluster_enabled:1
$ redis-cli -p 7000 INFO cluster_info | head -n 6
# Cluster Info
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0

The cluster fields in the other sections:

$ redis-cli -p 7000 INFO everything | grep -iE "cluster|link|slot_|migrat" | sort
# Cluster
# Cluster Info
cluster_connections:10
cluster_current_epoch:6
cluster_enabled:1
cluster_enabled:1
cluster_known_nodes:6
cluster_my_epoch:1
...
cluster_size:3
...
cluster_state:ok
...
cmdstat_cluster|info:calls=6,usec=283,usec_per_call=47.25,rejected_calls=0,failed_calls=0
...
mem_cluster_links:11680
mem_cluster_slot_export:0
mem_cluster_slot_import:0
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0
mem_cluster_slot_migration_output_buffer:0
migrate_cached_sockets:0
redis_mode:cluster
total_cluster_links_buffer_limit_exceeded:0
total_net_cluster_slot_export_bytes:0
total_net_cluster_slot_import_bytes:0

cluster_connections is the number of bus connections, and redis_mode:cluster tells a client it talks to a cluster node. Replication is in INFO replication. On a primary it lists its replicas with their offset and lag in seconds, and on a replica it gives the link status and the offsets:

$ redis-cli -p 7000 INFO replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=7003,state=online,offset=914085,lag=0
master_failover_state:no-failover
master_replid:bd114f8adce4b5fa852f03fb2e5f26c571d8f24c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:914085
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:67108864
repl_backlog_first_byte_offset:1
repl_backlog_histlen:914085
$ redis-cli -p 7003 INFO replication
# Replication
role:slave
master_host:127.0.0.1
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:914085
slave_repl_offset:914085
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:bd114f8adce4b5fa852f03fb2e5f26c571d8f24c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:914085
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:67108864
repl_backlog_first_byte_offset:1
repl_backlog_histlen:914085

(The output is shortened to the fields discussed here.)

The replica is caught up when slave_repl_offset equals the primary’s master_repl_offset. master_link_status:up and master_sync_in_progress:0 say the link is established and no full copy is running. INFO stats on a primary counts the syncs: sync_full, sync_partial_ok and sync_partial_err. A rising sync_full means replicas copy everything again: they fell out of the backlog (repl-backlog-size) or were restarted.

Memory a migration holds

Section titled “Memory a migration holds”

INFO memory has three fields for atomic slot migrations: mem_cluster_slot_migration_output_buffer is what the source holds for the migrations it exports, mem_cluster_slot_migration_input_buffer what a node that imports slots holds (the writes it keeps while the copy arrives, and input not parsed yet), and mem_cluster_slot_migration_input_buffer_peak the most the second one held. They are 0 when no migration is running, apart from the peak. With replica-full-sync-buffer-limit set on a target, the input buffer rises while the copy arrives. INFO stats counts the bytes a node exported for slot migrations in total_net_cluster_slot_export_bytes.

Slot statistics

Section titled “Slot statistics”

CLUSTER SLOT-STATS shows the keys, memory, CPU and network bytes of each slot, which is how you find the hot or heavy slots before you rebalance. See Scaling a cluster.

Metrics endpoint

Section titled “Metrics endpoint”

HotKV serves Prometheus metrics on the port set by metrics-port (restart only, bound to 127.0.0.1; see Monitoring). A cluster node serves the same series as any other node. None of them is about the cluster: nothing for the state, the nodes, the slots, the links or the migrations:

$ curl -s http://127.0.0.1:9000/metrics | grep -iE "cluster|slot|migrat"; echo exit=$?
exit=1

The replication series are there. On a replica they show the role, the replication offset and the lag, and on a primary the number of replicas:

$ curl -s http://127.0.0.1:9001/metrics | grep -E "^hotkv_replication"
hotkv_replication_role{role="replica"} 0
hotkv_replication_offset 911423
hotkv_replication_lag_microseconds 0
hotkv_replication_connected_replicas 0
$ redis-cli -p 7001 INFO replication | grep -E "^(role|master_repl_offset|slave_repl_offset|master_link_status)"
role:slave
master_link_status:up
slave_repl_offset:911423
master_repl_offset:911423
$ curl -s http://127.0.0.1:9000/metrics | grep -E "^hotkv_replication"
hotkv_replication_role{role="primary"} 1
hotkv_replication_offset 0
hotkv_replication_lag_microseconds 0
hotkv_replication_connected_replicas 1

So scrape the metrics port of each node for memory, clients, commands and replication, and for the cluster itself use a script that polls CLUSTER INFO and CLUSTER NODES and exposes the result in your monitoring system, as in the next section. A common way is to run the script from your monitoring agent and alert when it exits with 1.

A health-check script

Section titled “A health-check script”

This script exits with status 0 when the cluster looks healthy and 1 when it does not, and prints one line for each problem. It reads CLUSTER INFO, checks every node’s flags and bus link, asks every replica whether its link to its primary is up, requires that every primary that serves slots has at least MIN_REPLICAS replicas (default 1), and runs redis-cli --cluster check to catch open slots and nodes that disagree about the slots. Set REDISCLI_AUTH when the nodes need a password.

#!/usr/bin/env bash
# Usage: check-cluster.sh HOST:PORT
# Exits 0 when the cluster looks healthy and 1 when it does not, and prints what is wrong.
# Set REDISCLI_AUTH to the password when the nodes need one.
set -u
seed=${1:?usage: check-cluster.sh HOST:PORT}
min_replicas=${MIN_REPLICAS:-1}
rc() { timeout 5 redis-cli -h "${1%:*}" -p "${1##*:}" "${@:2}"; }
problems=0
bad() { echo "PROBLEM: $*"; problems=$((problems + 1)); }


info=$(rc "$seed" CLUSTER INFO 2>&1) || { echo "PROBLEM: cannot reach $seed: $info"; exit 1; }
field() { printf '%s\n' "$info" | tr -d '\r' | awk -F: -v k="$1" '$1 == k { print $2 }'; }


[ "$(field cluster_state)" = ok ] || bad "cluster_state is $(field cluster_state)"
[ "$(field cluster_slots_ok)" = 16384 ] || bad "only $(field cluster_slots_ok) of 16384 slots are ok"
for f in cluster_slots_pfail cluster_slots_fail cluster_nodes_pfail cluster_nodes_fail; do
  [ "$(field "$f")" = 0 ] || bad "$f is $(field "$f")"
done


declare -A replicas   # primary id -> number of replicas, for primaries that serve slots
while read -r id addr flags master _ _ _ link slots; do
  node=${addr%%@*}
  case $flags in *fail*|*noaddr*|*handshake*) bad "$node has flags $flags" ;; esac
  [ "$link" = connected ] || bad "$node: bus link is $link"
  case $flags in
    *master*) [ -n "$slots" ] && replicas[$id]=${replicas[$id]:-0} ;;
    *slave*)
      replicas[$master]=$(( ${replicas[$master]:-0} + 1 ))
      status=$(rc "$node" INFO replication 2>/dev/null | tr -d '\r' | awk -F: '$1 == "master_link_status" { print $2 }')
      [ "$status" = up ] || bad "replica $node: master_link_status is ${status:-unknown}" ;;
  esac
done < <(rc "$seed" CLUSTER NODES)


for id in "${!replicas[@]}"; do
  [ "${replicas[$id]}" -ge "$min_replicas" ] || bad "primary $id has ${replicas[$id]} replicas, want $min_replicas"
done


# open slots and slots that nodes disagree about
timeout 60 redis-cli --cluster check "$seed" > /dev/null 2>&1 || bad "redis-cli --cluster check reports a problem"


if [ "$problems" -eq 0 ]; then echo "OK: cluster_state ok, $(field cluster_known_nodes) nodes"; exit 0; fi
exit 1

Run healthy, it prints one line. With a replica frozen (kill -STOP) it names the replica:

$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
OK: cluster_state ok, 6 nodes
exit=0
$ kill -STOP <pid>
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cluster_nodes_fail is 1
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
PROBLEM: redis-cli --cluster check reports a problem
exit=1
$ kill -CONT <pid>
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
OK: cluster_state ok, 6 nodes
exit=0

A slot left open by a migration, and a slot that was removed on one node, are caught by the last check even though cluster_state stays ok:

127.0.0.1:7000> CLUSTER SETSLOT 100 MIGRATING 2479e739f79694cf9134902aecfb535a5bb50a54
OK
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: redis-cli --cluster check reports a problem
exit=1
127.0.0.1:7000> CLUSTER SETSLOT 100 STABLE
OK
127.0.0.1:7000> CLUSTER DELSLOTS 5
OK
$ ./check-cluster.sh 127.0.0.1:7001; echo exit=$?
PROBLEM: redis-cli --cluster check reports a problem
exit=1
127.0.0.1:7000> CLUSTER ADDSLOTS 5
OK

A replica that is down, and then a primary and its replica lost together, which leaves slots without an owner:

$ redis-cli -p 7003 SHUTDOWN NOSAVE
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cluster_nodes_fail is 1
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: 127.0.0.1:7003: bus link is disconnected
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
exit=1
$ kill -9 <pid> <pid>
$ ./check-cluster.sh 127.0.0.1:7001; echo exit=$?
PROBLEM: cluster_state is fail
PROBLEM: only 10923 of 16384 slots are ok
PROBLEM: cluster_slots_fail is 5461
PROBLEM: cluster_nodes_fail is 2
PROBLEM: 127.0.0.1:7003 has flags slave,fail
PROBLEM: 127.0.0.1:7003: bus link is disconnected
PROBLEM: replica 127.0.0.1:7003: master_link_status is unknown
PROBLEM: 127.0.0.1:7000 has flags master,fail
PROBLEM: 127.0.0.1:7000: bus link is disconnected
PROBLEM: redis-cli --cluster check reports a problem
exit=1
$ ./check-cluster.sh 127.0.0.1:7000; echo exit=$?
PROBLEM: cannot reach 127.0.0.1:7000: Could not connect to Redis at 127.0.0.1:7000: Connection refused
exit=1

Point the script at a node that is not the one you suspect, and at a second node if the first does not answer: a monitor that asks only one node is blind when that node is the one that failed. Run it every minute from cron or from your monitoring agent, and alert on exit status 1.

Alert conditions

Section titled “Alert conditions”

These are suggestions: thresholds are yours to set, and each is tied to a field you can read with the commands above.

Alert on Field or command Why
The cluster is down cluster_state is not ok on a node, for more than a few seconds Keyed commands fail on that node.
Slots without a healthy owner cluster_slots_ok below 16384, or cluster_slots_fail above 0 A shard is unavailable.
A node failing cluster_nodes_fail above 0, or a fail flag in CLUSTER NODES, for more than a minute A node is down and has not come back. Page for a primary without a replica.
A node suspected for long cluster_nodes_pfail above 0 across several checks A node answers badly, which often precedes a failover.
A replica not following master_link_status:down in INFO replication on a replica, or fewer connected_slaves than expected on a primary The shard has no spare copy.
A replica far behind slave_repl_offset well below the primary’s master_repl_offset, or lag in the primary’s slave0 line A failover now would lose that much.
Replicas copying everything again sync_full in INFO stats rising The replica fell out of the backlog or keeps restarting.
A failover happened cluster_current_epoch rose, or the primary of a shard changed in CLUSTER NODES Look at why the old primary stopped.
A node with the wrong number of replicas The script’s replica count below MIN_REPLICAS One more failure takes a shard down.
An open or inconsistent slot redis-cli --cluster check exits with 1 A reshard did not finish (see Scaling).
A stuck migration cluster_slot_migration_active_tasks above 0 for longer than your largest move takes, or a failed state in CLUSTER GETSLOTMIGRATIONS See Scaling.
Bus trouble total_cluster_links_buffer_limit_exceeded rising, or send-buffer-allocated growing in CLUSTER LINKS A node does not keep up with the bus.
Memory used_memory against maxmemory on every primary A full primary answers OOM, and a move onto it fails.

Next steps

Section titled “Next steps”