Replicas and failover

Last updated 12 Oct 2026

Not on HotKV CloudOn-Premise: no license needed

A replica keeps a live copy of one primary and takes over its slots when that primary fails. HotKV has no Sentinel: in cluster mode the nodes watch each other and promote a replica themselves, so a cluster with replicas needs no extra service to stay available. This page covers how many replicas to run, how to add and use them, what a failover looks like from the outside, the ways to start one yourself, and the places where a write can still be lost.

The examples use a cluster on 127.0.0.1 with three primaries and three replicas on ports 8100 to 8105, started with --appendonly yes, unless a section says otherwise. The partition example uses two hosts, 10.0.0.1 and 10.0.0.2.

How many replicas

Replicas per primary	What a failed primary means	Notes
0	Its slots have no owner until the node is back. With the default settings the whole cluster answers `CLUSTERDOWN`.	Data comes back with the node if it keeps an AOF or snapshots.
1	The replica takes over by itself.	Until the old primary is back or you add a replica, the shard has no spare copy.
2	The best replica takes over and the shard still has a replica.	A shard with spare replicas can lend one to a shard that has none (see Replica migration).

Start with one replica per primary and put it on a different host from its primary, in a different zone if you can. A replica on the same machine protects against a crash of the process, not of the machine. Replicas do not add write capacity: every write still goes to the primary. They add read capacity only for clients that ask for replica reads (see Reads from replicas).

This is what a cluster of three primaries with no replicas does when one primary dies, with cluster-node-timeout 2000 and data written with an AOF:

$ kill -9 <pid>
$ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail"
cluster_state:fail
cluster_slots_ok:10923
cluster_slots_fail:5461
127.0.0.1:8100> GET {cart1}:a
(error) CLUSTERDOWN The cluster is down

Two settings change that. With cluster-allow-reads-when-down yes a node keeps answering reads for the slots it owns, and writes still fail. With cluster-require-full-coverage no the primaries that are left keep serving their own slots in full, and only the keys of the missing slots fail:

127.0.0.1:8100> CONFIG SET cluster-allow-reads-when-down yes
OK
127.0.0.1:8100> GET {cart1}:a
"1"
127.0.0.1:8100> GET {user2}:a
(error) MOVED 12233 127.0.0.1:8102
127.0.0.1:8100> SET {cart1}:a 10
(error) CLUSTERDOWN The cluster is down and only accepts read commands
127.0.0.1:8100> CONFIG SET cluster-allow-reads-when-down no
OK
127.0.0.1:8100> CONFIG SET cluster-require-full-coverage no
OK
127.0.0.1:8101> CONFIG SET cluster-require-full-coverage no
OK
$ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail"
cluster_state:ok
cluster_slots_ok:10923
cluster_slots_fail:5461
127.0.0.1:8100> SET {cart1}:a 10
OK

Leave cluster-require-full-coverage on unless losing a shard should not stop the others: with it off, an application that expects a multi-key operation to work across shards sees failures only for the keys that are gone. Starting the dead node again brings the slots and its data back:

$ hotkv --port 8102 --cluster-enabled yes --dir node-8102 --daemonize yes --logfile hotkv.log --appendonly yes --cluster-node-timeout 2000
$ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail"
cluster_state:ok
cluster_slots_ok:16384
cluster_slots_fail:0
$ redis-cli -c -p 8100 GET "{user2}:a"
3

Add a replica

A new replica must be empty. Create the cluster with replicas in one step with redis-cli --cluster create ... --cluster-replicas 1, or add one to a running cluster with redis-cli --cluster add-node ... --cluster-slave as shown in Scaling a cluster. Both end in CLUSTER REPLICATE, which you can also send to a node that has already joined:

$ redis-cli --cluster add-node 127.0.0.1:8106 127.0.0.1:8100 | tail -n 1
[OK] New node added correctly.
127.0.0.1:8106> CLUSTER REPLICATE c5bcbe648892246dde31b5d78443297b493afb63
OK
127.0.0.1:8106> ROLE
1) "slave"
2) "127.0.0.1"
3) (integer) 8100
4) "connected"
5) (integer) 105
$ redis-cli -p 8100 CLUSTER NODES | grep c5bcbe648892246dde31b5d78443297b493afb63 | cut -d" " -f2,3,4 | sort
127.0.0.1:8100@18100 myself,master -
127.0.0.1:8105@18105 slave c5bcbe648892246dde31b5d78443297b493afb63
127.0.0.1:8106@18106 slave c5bcbe648892246dde31b5d78443297b493afb63

The replica copies the primary’s data first. While that full copy arrives it answers commands that read or write data with LOADING (PING and INFO still answer). After a short break in the link, a replica continues from where it stopped instead of copying everything again.

CLUSTER REPLICATE NO ONE turns a replica back into an empty primary of a shard of its own. It drops the replica’s data and gives it no slots:

127.0.0.1:8106> CLUSTER REPLICATE NO ONE
OK
127.0.0.1:8106> ROLE
1) "master"
2) (integer) 123
3) (empty array)

A node that is a primary with slots or keys cannot become a replica of another node: move its slots away first (see Scaling a cluster). Replicas connect to the client port of their primary: a cluster node needs only its client port and its bus port open between hosts. A replica logs in with masterauth when it is set; with the same requirepass on every node nothing more is needed.

Reads from replicas

By default a replica redirects every command about a key to the primary with -MOVED. A client that sends READONLY on its connection may read from the replica instead; READWRITE goes back. Writes are never accepted by a replica:

127.0.0.1:8105> GET {cart1}:a
(error) MOVED 3311 127.0.0.1:8100
$ printf "READONLY\nGET {cart1}:a\nSET {cart1}:a 2\nREADWRITE\nGET {cart1}:a\n" | redis-cli --no-raw -p 8105
OK
"1"
(error) MOVED 3311 127.0.0.1:8100
OK
(error) MOVED 3311 127.0.0.1:8100

Most cluster clients have a switch for this (see Clients). A read from a replica can be stale: replication is asynchronous, so a replica may not have applied a write the primary acknowledged a moment ago. Read from the primary when a client must see its own latest write.

Replication is asynchronous

A primary acknowledges a write before its replicas have it. WAIT lets a client wait until a number of replicas have the writes it made, and returns how many do, up to the timeout in milliseconds. WAITAOF waits for the writes to be in the append-only file on disk, on this node and on a number of replicas, and returns both counts:

$ printf "SET {cart1}:w 1\nWAIT 1 100\nWAIT 2 100\nWAITAOF 1 1 100\nWAITAOF 1 2 100\n" | redis-cli --no-raw -p 8100
OK
(integer) 1
(integer) 1
1) (integer) 0
2) (integer) 0
1) (integer) 0
2) (integer) 0

WAIT 2 found one replica, because the shard has one. WAITAOF returned zeros here because the timeout of 100 ms ran out before the AOF reached the disk: with appendfsync everysec that takes up to a second. With a longer timeout, or with WAITAOF 1 0 0, which waits for the local AOF, both counts came back as 1:

127.0.0.1:8100> SET {cart1}:t 2
OK
127.0.0.1:8100> WAITAOF 1 1 2000
1) (integer) 1
2) (integer) 1

With the replica frozen, WAIT reports that the write did not reach it, and the write stays in place on the primary:

$ kill -STOP <pid>
$ printf "SET {cart1}:w 2\nWAIT 1 300\nWAITAOF 1 1 300\n" | redis-cli --no-raw -p 8100
OK
(integer) 0
1) (integer) 0
2) (integer) 0
$ kill -CONT <pid>

WAIT narrows the window in which a failover loses a write but does not close it (see Where writes can be lost).

Automatic failover

When a primary stops answering, the cluster does this without an operator:

After cluster-node-timeout (default 15000 ms) of silence, each node flags the primary PFAIL (it shows as fail? in CLUSTER NODES).
When a majority of the primaries that serve slots report it, the node flags it FAIL and tells the others.
Its replica with the most data asks the primaries for votes. With a majority it becomes the primary, takes the slots and announces it. A shard with more than one replica ranks them by how much data each has.
The other nodes update their maps. Clients get MOVED to the new primary. The shard’s other replicas follow the new primary, and the old primary rejoins as a replica of the new one when it comes back.

A replica whose data is too old (older than cluster-node-timeout times cluster-replica-validity-factor, plus the replication heartbeat period, 10 seconds by default; 0 turns off this age check only), one that has never finished a sync, and one with cluster-replica-no-failover yes do not start an automatic failover. A shard without such a replica stays down until its primary returns or you promote a replica yourself.

What it looks like

The drill at the end of this page (see the drill) kills a primary with kill -9 while writer.py writes numbered keys, and prints a line each time the picture changes on a surviving node: the cluster state, the flags of the old primary and the role of its replica. With the default timeout the old primary goes from master to master,fail? (suspected) to master,fail (agreed), and the replica from slave to master.

The shard is unavailable for about the node timeout plus a few seconds: the default 15 seconds of silence before the nodes suspect the primary, a few more for the majority to agree, and a fraction of a second for the election. The client gets ConnectionError for every write to that shard until the new primary takes over, then carries on. The drill’s last line also shows the price of asynchronous replication: the writes the primary acknowledged but the replica had not received when it was killed are gone, and are counted as missing.

The log of the replica that won, and of one of the nodes that voted:

--- log of 8103 (the replica)
[hotkv] [info] Cluster: node 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it
[hotkv] [warning] FAIL message received from 9f94ae9ebededa10d6b070be45745b51916ff313 (127.0.0.1:8104) about 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102)
[hotkv] [info] This is the best ranked replica and can initiate the election immediately.
[hotkv] [info] Cluster: failover election for epoch 7: votes asked of the primaries
[hotkv] [info] Cluster: Failover vote 1/3 (need 2)
[hotkv] [info] Cluster: Failover vote 2/3 (need 2)
[hotkv] [info] Cluster: this replica won the failover election: it takes over as the primary
[hotkv] [info] Cluster: Completing failover
[hotkv] [info] Replication: cluster failover - becoming primary (accepting replicas)
--- log of 8101 (a voter)
[hotkv] [info] Cluster: node 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it
[hotkv] [warning] FAIL message received from 9f94ae9ebededa10d6b070be45745b51916ff313 (127.0.0.1:8104) about 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102)
[hotkv] [info] Cluster: voted for 4ccb024649e2904d33704bae0c3f3ba8fb149909 (127.0.0.1:8103) in the failover election of epoch 7

With --cluster-node-timeout 2000 the same sequence takes a little longer than the timeout instead of about 20 seconds.

Choose the node timeout

cluster-node-timeout is the length of silence after which a node counts another as failing, so it sets how long a real failure takes to notice and how patient the cluster is with a node that is only slow. A short timeout recovers faster, and also promotes a replica for a node that merely paused: an overloaded host, a stalled disk or a network blip longer than the timeout looks like a failure. Pick a value larger than the longest pause you see on your hosts and network. CONFIG SET changes it at runtime; set it to the same value on every node.

Watching and restarting

The old primary can be started again at any time with its own data directory and configuration. It finds out that another node took its slots, and becomes a replica of the new primary, then catches up:

$ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort
127.0.0.1:8100@18100 slave 4ae1e9d673d4d1f67fc15ac09987f85a96079ae7 connected
127.0.0.1:8101@18101 myself,master - connected 5461-10922
127.0.0.1:8102@18102 master - connected 10923-16383
127.0.0.1:8103@18103 slave f31014c5d868f136d477f68a908ae05a55800092 connected
127.0.0.1:8104@18104 master - connected 0-5460
127.0.0.1:8105@18105 slave 00cc163c39e228a537dbc06b5601bfa7f3cb781b connected

If the old primary comes back without its data (an empty directory), do not let it rejoin with the old ID before you check the roles: see Backup and restore.

Network partitions

A node cut off from most of the cluster must stop accepting writes, because the majority will elect a new primary for its slots. HotKV does this: a node that cannot reach a majority of the primaries marks the cluster as down and refuses keys. The example uses two hosts: five nodes on 10.0.0.1 (primaries 8500 and 8501, replicas 8503, 8504 and 8505) and one on 10.0.0.2 (the primary 8502, whose replica 8505 is on the other host), all with cluster-node-timeout 5000:

$ hotkv --port 8500 --bind 10.0.0.1 --cluster-enabled yes --protected-mode no --dir node-8500 --daemonize yes --logfile hotkv.log --cluster-node-timeout 5000 --appendonly yes

(Start the nodes on 10.0.0.1 this way on ports 8500, 8501, 8503, 8504 and 8505, and the one on 10.0.0.2 with --bind 10.0.0.2 on port 8502.) To simulate a partition, cut the nodes on 10.0.0.1 off from 10.0.0.2 on the client and bus ports by dropping the packets it sends:

$ sudo iptables -I INPUT -p tcp -s 10.0.0.2 -m multiport --dports 8500:8599,18500:18599 -j DROP
$ sudo iptables -I INPUT -p tcp -s 10.0.0.2 -m multiport --sports 8500:8599,18500:18599 -j DROP

(Remove the rules afterwards with the same commands and -D in place of -I.) Suppose a client writes {session1} keys straight to the isolated primary on 10.0.0.2 throughout. After 5 seconds the nodes on the isolated side flag the others PFAIL, and then:

--- CLUSTER NODES seen from the majority (10.0.0.1)
10.0.0.2:8502@18502 master,fail - disconnected
10.0.0.1:8500@18500 myself,master - connected 0-5460
10.0.0.1:8501@18501 master - connected 10923-16383
10.0.0.1:8503@18503 slave c88860a72dce6c55c08f09749533856e7dca2c27 connected
10.0.0.1:8504@18504 slave d8e3515ab2d8888ce9595e5f4099cc3b0996f2a5 connected
10.0.0.1:8505@18505 master - connected 5461-10922
--- CLUSTER NODES seen from the minority node (10.0.0.2)
10.0.0.2:8502@18502 myself,master - connected 5461-10922
10.0.0.1:8500@18500 master,fail? - disconnected 0-5460
10.0.0.1:8501@18501 master,fail? - disconnected 10923-16383
10.0.0.1:8503@18503 slave,fail? c88860a72dce6c55c08f09749533856e7dca2c27 disconnected
10.0.0.1:8504@18504 slave,fail? d8e3515ab2d8888ce9595e5f4099cc3b0996f2a5 disconnected
10.0.0.1:8505@18505 slave,fail? c78599fe62b4ee129b6e93e3032646aea0fce49e disconnected

The majority side promotes the replica 8505 and keeps serving. The minority node logs that it is alone, and from then on answers CLUSTERDOWN:

Cluster: the cluster is down: this node is in a minority partition, reaching no majority of the primaries that serve slots

The isolated primary keeps accepting writes for a while after the cut, because it needs the node timeout to notice. Those writes go nowhere: after the network is restored, the old primary becomes a replica of 8505 and the writes it acknowledged during the window are gone. A lower cluster-node-timeout shortens this window and a higher one stretches it. The rule is the same as for a failure: only the side with a majority of the primaries keeps serving, so run at least three primaries and spread them over hosts or zones so that no single host holds a majority.

Start a failover yourself

CLUSTER FAILOVER on a replica promotes it. Use it to move a primary to another host before maintenance, or to recover when automatic failover cannot run. There are three modes.

Mode	What it does	Use it when
`CLUSTER FAILOVER`	The primary pauses writes, the replica catches up with every write, then wins an election. No acknowledged write is lost.	The primary is up and you choose the moment.
`CLUSTER FAILOVER FORCE`	Skips the handshake with the primary and runs the election straight away. A majority of the primaries must still vote for it.	The primary is down or stuck but the cluster has a majority.
`CLUSTER FAILOVER TAKEOVER`	Takes over at once with a new epoch, without an election.	A majority of the primaries cannot be reached.

Every mode must be sent to a replica; sent to a primary it replies ERR You should send CLUSTER FAILOVER to a replica.

Start one failover at a time and wait until the replica has become a primary (ROLE, or CLUSTER NODES) before you start the next. Elections that begin at the same moment in different shards can collide on the same epoch, and the replica that loses never takes over, although CLUSTER FAILOVER answered OK: its log says Manual failover timed out. It did not win an election in time. If that happens, check the roles and send the command again.

The default mode

The default mode needs the primary to answer. It pauses the writes of its slots for a moment, so clients wait instead of failing, and no acknowledged write is lost:

127.0.0.1:8103> CLUSTER FAILOVER
OK
$ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort
127.0.0.1:8100@18100 slave 3bfee53e12421ac6c914bf03042dc2a2fde68fb4 connected
127.0.0.1:8101@18101 myself,master - connected 5461-10922
127.0.0.1:8102@18102 master - connected 10923-16383
127.0.0.1:8103@18103 master - connected 0-5460
127.0.0.1:8104@18104 slave 147c5200dfb75e7fcbe095c22dc93cc590cb7cac connected
127.0.0.1:8105@18105 slave f90cb48df290ae2ef2328925012abcebd48204e1 connected

The replica logs that the primary was paused, then that it caught up with the primary and is starting the election, and wins it. The default mode has cluster-manual-failover-timeout (default 5000 ms) to finish. If the primary does not answer, it gives up and logs it, and the replica stays a replica.

FORCE

With the primary frozen (kill -STOP), the default mode fails, and FORCE works:

$ kill -STOP <pid>
127.0.0.1:8104> CLUSTER FAILOVER
OK
$ redis-cli -p 8104 ROLE | head -n 1
slave
Manual failover timed out. It did not catch up with primary 216cb28fed6439091f6ca74f31c6a4a54bb7e7ed in time (CLUSTER FAILOVER FORCE or TAKEOVER override this)
127.0.0.1:8104> CLUSTER FAILOVER FORCE
OK
$ redis-cli -p 8104 ROLE | head -n 1
master
$ kill -CONT <pid>

The first reply is OK because the failover started, and the ROLE after it still says slave once the timeout has run out. The log line is what the replica writes when the timeout runs out. After FORCE the replica becomes the primary within a couple of seconds. When the frozen primary is resumed it finds it has lost its slots and turns into a replica. Writes it accepted after the last one the replica had are lost.

TAKEOVER

When most primaries are unreachable, nobody can vote, and FORCE runs out of time. TAKEOVER does not need a vote. Here two of the three primaries are frozen:

$ kill -STOP <pid> <pid>
127.0.0.1:8104> CLUSTER FAILOVER FORCE
OK
$ redis-cli -p 8104 ROLE | head -n 1
slave
Manual failover timed out. It did not win an election in time
127.0.0.1:8104> CLUSTER FAILOVER TAKEOVER
OK
$ redis-cli -p 8104 ROLE | head -n 1
master
$ kill -CONT <pid> <pid>

Use TAKEOVER only when you know the old primary will not be reached by clients that keep writing to it: it takes the slots without asking the cluster. When the other nodes come back, the node with the newer epoch wins and the old primary becomes its replica, losing what it had that the replica did not. A replica that never finished its first sync serves the shard with incomplete data after a takeover, so check it first.

CLUSTER FAILOVER with an unknown word fails with syntax error.

Planned restarts

To restart or upgrade a primary without a pause for its clients, hand its slots to a replica first and then stop it. The commands do it in one step.

SHUTDOWN SAFE refuses on a primary that serves slots, because its slots would be down until a replica took them:

127.0.0.1:8100> SHUTDOWN SAFE
(error) ERR Errors trying to SHUTDOWN. Check logs.
SHUTDOWN SAFE: this primary serves slots, which would be down until a replica took them over; not shutting down

SHUTDOWN FAILOVER hands the slots to a replica that has every write, and then stops. Clients see no error:

127.0.0.1:8100> SHUTDOWN FAILOVER
$ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort
127.0.0.1:8100@18100 master - disconnected
127.0.0.1:8101@18101 myself,master - connected 5461-10922
127.0.0.1:8102@18102 master - connected 10923-16383
127.0.0.1:8103@18103 master - connected 0-5460
127.0.0.1:8104@18104 slave 791b47c22caa971b3ae8cef340313db0480b135c connected
127.0.0.1:8105@18105 slave 24f55d649c24000381721f4691bd432915b9bfe2 connected

(The old primary shows as disconnected for a moment, until the others drop its entry as it is now the replica of 8103.) Start the node again and it comes back as a replica. SHUTDOWN SAFE on a replica, or on a primary without slots, just stops it. SHUTDOWN FAILOVER outside cluster mode is refused with ERR SHUTDOWN FAILOVER is only supported in cluster mode.

To get the same from systemctl stop or a SIGTERM, set shutdown-on-sigterm failover (and shutdown-on-sigint for SIGINT):

$ redis-cli -p 8100 CONFIG GET shutdown-on-sigterm
shutdown-on-sigterm
failover
$ kill -TERM <pid>

If no replica is ready, a primary logs Unable to find a replica to perform the auto failover on shutdown. and stops; the cluster then promotes a replica as for any other failure.

Replica migration

When a primary loses its last replica, a spare replica of another shard can move to it, so a second failure does not take the shard down. It is on by default (cluster-allow-replica-migration yes), and a shard only lends a replica while it keeps more than cluster-migration-barrier (default 1) of them. Take a cluster of three primaries with two replicas each (nodes 8100 to 8108). When both replicas of the primary on port 8101, nodes 8107 and 8108, are killed with kill -9, the CLUSTER NODES listing a few seconds later shows a replica of another shard moving over:

$ redis-cli -p 8100 CONFIG GET cluster-migration-barrier
cluster-migration-barrier
1
$ redis-cli -p 8100 CLUSTER NODES | awk "{print \$2, \$3, \$4}" | cut -d@ -f2- | sort
18100 myself,master -
18101 master -
18102 master -
18103 slave 58f6d7f3fce5bdee819d27a880087133755fac78
18104 slave 58f6d7f3fce5bdee819d27a880087133755fac78
18105 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9
18106 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9
18107 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52
18108 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52
$ redis-cli -p 8100 CLUSTER NODES | awk "{print \$2, \$3, \$4}" | cut -d@ -f2- | sort
18100 myself,master -
18101 master -
18102 master -
18103 slave 58f6d7f3fce5bdee819d27a880087133755fac78
18104 slave 58f6d7f3fce5bdee819d27a880087133755fac78
18105 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52
18106 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9
18107 slave,fail 487a60dfb7ae5e1379913a57adfbdbba95a72e52
18108 slave,fail 487a60dfb7ae5e1379913a57adfbdbba95a72e52
Migrating to orphaned primary: replica 352ae1618d3f148c8e31ef6a44582b8803887af8 moves to 487a60dfb7ae5e1379913a57adfbdbba95a72e52 (127.0.0.1:8101) of shard fcb3d0a02

Node 8105, which followed another primary, now follows the primary of 8101. The last line is from the log of node 8105. The move waits a few seconds after the loss, so that the shard’s own replicas have time to report. Set cluster-allow-replica-migration no if you place replicas by hand and do not want them to move.

Keep a replica from taking over

A replica with cluster-replica-no-failover yes shows the flag nofailover in CLUSTER NODES, is left out of the ranking and never starts an automatic failover. Use it for a replica you keep for backups or reads on a weaker host. CLUSTER FAILOVER still promotes it when you ask. When it was the only replica and the primary died, nothing promoted it until CLUSTER FAILOVER FORCE:

127.0.0.1:8104> CONFIG SET cluster-replica-no-failover yes
OK
$ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3 | grep slave
127.0.0.1:8104@18104 slave,nofailover
127.0.0.1:8105@18105 slave
127.0.0.1:8103@18103 slave
$ kill -9 <pid>
$ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,9 | sort
127.0.0.1:8100@18100 master,fail - 0-5460
...
127.0.0.1:8104@18104 slave,nofailover e25c9fc75c319b0b26c52a7c3f461e7ecd74bd01
...
$ redis-cli -p 8101 CLUSTER INFO | grep cluster_state
cluster_state:fail
127.0.0.1:8104> CLUSTER FAILOVER FORCE
OK
$ redis-cli -p 8101 CLUSTER INFO | grep cluster_state
cluster_state:ok

Where writes can be lost

Replication is asynchronous, so a write the primary acknowledged can be lost in the cases below. The list is about replication and failover. It leaves out what any server does: eviction and expiry remove keys, and a node that crashes and restarts from its own files loses the writes its append-only file had not flushed yet (about a second with appendfsync everysec). The cases are:

The primary fails before its replica has the write. A failover promotes the replica, which does not have it. Only the few writes in flight at that moment are affected. WAIT 1 timeout after a write makes the client wait until a replica has it, and shrinks the window to the time between the write and the WAIT reply; a client that did not get the reply must treat the write as unknown.
A primary is cut off from the majority. It keeps acknowledging writes until it notices, up to about the node timeout, and loses them when it rejoins as a replica (see Network partitions).
FORCE or TAKEOVER on a replica that is behind, or a primary that was frozen and keeps its late writes.
A restart without data. A primary that comes back with an empty directory has nothing to give, and its replicas copy the emptiness. Check the roles before you start a node that lost its disk.

The planned ways of moving a primary (CLUSTER FAILOVER, SHUTDOWN FAILOVER) lose no acknowledged write: the primary stops accepting writes, the replica catches up, and only then does it take over.

A failover drill

Run this on a practice cluster that lives on one host, such as the one from the quick start, before you trust a cluster in production. It needs writer.py from Scaling a cluster in the current directory and the redis package (install it in a virtual environment, as shown in Scaling a cluster). It picks a primary that has a replica, kills it, and prints a timeline of the failover.

"""Kill one primary of a practice cluster on this host while writer.py runs, and report what happened.

Usage: python3 drill.py HOST:PORT      (writer.py must be in the current directory)
"""
import os
import signal
import subprocess
import sys
import time

import redis


def connect(addr):
    host, port = addr.rsplit(":", 1)
    return redis.Redis(host=host, port=int(port), decode_responses=True)


def nodes(client):
    result = []
    for line in client.execute_command("CLUSTER", "NODES").splitlines():
        f = line.split()
        result.append({"id": f[0], "addr": f[1].split("@")[0], "flags": f[2].replace("myself,", ""),
                       "master": f[3], "slots": f[8:]})
    return result


seed = connect(sys.argv[1])
cluster = nodes(seed)
victim = next(n for n in cluster if n["flags"] == "master" and n["slots"]
              and any(r["master"] == n["id"] for r in cluster))
replica = next(r for r in cluster if r["master"] == victim["id"])
survivor = next(n for n in cluster if n["flags"] == "master" and n["slots"] and n is not victim)
pid = connect(victim["addr"]).info("server")["process_id"]

writer = subprocess.Popen([sys.executable, "writer.py", sys.argv[1], "60"],
                          stdout=open("drill-writer.out", "w"), stderr=subprocess.STDOUT)
time.sleep(5)
print(f"primary {victim['addr']} (pid {pid}), replica {replica['addr']}")
start = time.time()
os.kill(pid, signal.SIGKILL)
print(f"{0:5.1f}s kill -9 {pid}")

watch, last = connect(survivor["addr"]), None
while time.time() - start < 120:
    try:
        state = watch.execute_command("CLUSTER", "INFO").split("cluster_state:")[1].split()[0]
        flags = {n["addr"]: n["flags"] for n in nodes(watch)}
        view = (state, flags[victim["addr"]], flags[replica["addr"]])
    except redis.RedisError:
        time.sleep(0.1)
        continue
    if view != last:
        print(f"{time.time() - start:5.1f}s cluster_state={view[0]}  "
              f"old primary: {view[1]}  replica: {view[2]}")
        last = view
    if view[0] == "ok" and view[2] == "master":
        break
    time.sleep(0.1)

writer.wait()
print(open("drill-writer.out").read().strip().splitlines()[-1])

Read the last line: errors and longest_stall are what your clients lived through, and missing is the number of acknowledged writes that were lost. Start the killed node again afterwards (it comes back as a replica), and repeat the drill with your real client library in place of writer.py to see what your application does.

Next steps

Monitoring a cluster for the alerts to set on failover and replication lag.
Upgrades for the planned way to replace every node.
Scaling a cluster to add replicas and move slots.
Replication for replication outside cluster mode.

Ask the HotKV docs