# Replicas and failover > How many replicas a cluster needs, how replicas serve reads, what automatic and manual failover do and how long they take, and where writes can be lost. URL: https://docs.hotkv.com/clustering/replicas-and-failover/ **Availability:** Not on HotKV Cloud. On-Premise: no license needed. A replica keeps a live copy of one primary and takes over its slots when that primary fails. HotKV has no Sentinel: in cluster mode the nodes watch each other and promote a replica themselves, so a cluster with replicas needs no extra service to stay available. This page covers how many replicas to run, how to add and use them, what a failover looks like from the outside, the ways to start one yourself, and the places where a write can still be lost. The examples use a cluster on `127.0.0.1` with three primaries and three replicas on ports 8100 to 8105, started with `--appendonly yes`, unless a section says otherwise. The partition example uses two hosts, `10.0.0.1` and `10.0.0.2`. ## How many replicas | Replicas per primary | What a failed primary means | Notes | | --- | --- | --- | | 0 | Its slots have no owner until the node is back. With the default settings the whole cluster answers `CLUSTERDOWN`. | Data comes back with the node if it keeps an AOF or snapshots. | | 1 | The replica takes over by itself. | Until the old primary is back or you add a replica, the shard has no spare copy. | | 2 | The best replica takes over and the shard still has a replica. | A shard with spare replicas can lend one to a shard that has none (see [Replica migration](#replica-migration)). | Start with one replica per primary and put it on a different host from its primary, in a different zone if you can. A replica on the same machine protects against a crash of the process, not of the machine. Replicas do not add write capacity: every write still goes to the primary. They add read capacity only for clients that ask for replica reads (see [Reads from replicas](#reads-from-replicas)). This is what a cluster of three primaries with no replicas does when one primary dies, with `cluster-node-timeout 2000` and data written with an AOF: ``` $ kill -9 $ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail" cluster_state:fail cluster_slots_ok:10923 cluster_slots_fail:5461 127.0.0.1:8100> GET {cart1}:a (error) CLUSTERDOWN The cluster is down ``` Two settings change that. With `cluster-allow-reads-when-down yes` a node keeps answering reads for the slots it owns, and writes still fail. With `cluster-require-full-coverage no` the primaries that are left keep serving their own slots in full, and only the keys of the missing slots fail: ``` 127.0.0.1:8100> CONFIG SET cluster-allow-reads-when-down yes OK 127.0.0.1:8100> GET {cart1}:a "1" 127.0.0.1:8100> GET {user2}:a (error) MOVED 12233 127.0.0.1:8102 127.0.0.1:8100> SET {cart1}:a 10 (error) CLUSTERDOWN The cluster is down and only accepts read commands 127.0.0.1:8100> CONFIG SET cluster-allow-reads-when-down no OK 127.0.0.1:8100> CONFIG SET cluster-require-full-coverage no OK 127.0.0.1:8101> CONFIG SET cluster-require-full-coverage no OK $ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail" cluster_state:ok cluster_slots_ok:10923 cluster_slots_fail:5461 127.0.0.1:8100> SET {cart1}:a 10 OK ``` Leave `cluster-require-full-coverage` on unless losing a shard should not stop the others: with it off, an application that expects a multi-key operation to work across shards sees failures only for the keys that are gone. Starting the dead node again brings the slots and its data back: ``` $ hotkv --port 8102 --cluster-enabled yes --dir node-8102 --daemonize yes --logfile hotkv.log --appendonly yes --cluster-node-timeout 2000 $ redis-cli -p 8100 CLUSTER INFO | grep -E "cluster_state|cluster_slots_ok|cluster_slots_fail" cluster_state:ok cluster_slots_ok:16384 cluster_slots_fail:0 $ redis-cli -c -p 8100 GET "{user2}:a" 3 ``` ## Add a replica A new replica must be empty. Create the cluster with replicas in one step with `redis-cli --cluster create ... --cluster-replicas 1`, or add one to a running cluster with `redis-cli --cluster add-node ... --cluster-slave` as shown in [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md#add-a-node). Both end in [`CLUSTER REPLICATE`](https://docs.hotkv.com/commands/cluster-replicate/index.md), which you can also send to a node that has already joined: ``` $ redis-cli --cluster add-node 127.0.0.1:8106 127.0.0.1:8100 | tail -n 1 [OK] New node added correctly. 127.0.0.1:8106> CLUSTER REPLICATE c5bcbe648892246dde31b5d78443297b493afb63 OK 127.0.0.1:8106> ROLE 1) "slave" 2) "127.0.0.1" 3) (integer) 8100 4) "connected" 5) (integer) 105 $ redis-cli -p 8100 CLUSTER NODES | grep c5bcbe648892246dde31b5d78443297b493afb63 | cut -d" " -f2,3,4 | sort 127.0.0.1:8100@18100 myself,master - 127.0.0.1:8105@18105 slave c5bcbe648892246dde31b5d78443297b493afb63 127.0.0.1:8106@18106 slave c5bcbe648892246dde31b5d78443297b493afb63 ``` The replica copies the primary's data first. While that full copy arrives it answers commands that read or write data with `LOADING` (`PING` and `INFO` still answer). After a short break in the link, a replica continues from where it stopped instead of copying everything again. `CLUSTER REPLICATE NO ONE` turns a replica back into an empty primary of a shard of its own. It drops the replica's data and gives it no slots: ``` 127.0.0.1:8106> CLUSTER REPLICATE NO ONE OK 127.0.0.1:8106> ROLE 1) "master" 2) (integer) 123 3) (empty array) ``` A node that is a primary with slots or keys cannot become a replica of another node: move its slots away first (see [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md)). Replicas connect to the **client port** of their primary: a cluster node needs only its client port and its bus port open between hosts. A replica logs in with `masterauth` when it is set; with the same `requirepass` on every node nothing more is needed. ## Reads from replicas By default a replica redirects every command about a key to the primary with `-MOVED`. A client that sends `READONLY` on its connection may read from the replica instead; `READWRITE` goes back. Writes are never accepted by a replica: ``` 127.0.0.1:8105> GET {cart1}:a (error) MOVED 3311 127.0.0.1:8100 $ printf "READONLY\nGET {cart1}:a\nSET {cart1}:a 2\nREADWRITE\nGET {cart1}:a\n" | redis-cli --no-raw -p 8105 OK "1" (error) MOVED 3311 127.0.0.1:8100 OK (error) MOVED 3311 127.0.0.1:8100 ``` Most cluster clients have a switch for this (see [Clients](https://docs.hotkv.com/clustering/clients/index.md)). A read from a replica can be **stale**: replication is asynchronous, so a replica may not have applied a write the primary acknowledged a moment ago. Read from the primary when a client must see its own latest write. ## Replication is asynchronous A primary acknowledges a write before its replicas have it. [`WAIT`](https://docs.hotkv.com/commands/wait/index.md) lets a client wait until a number of replicas have the writes it made, and returns how many do, up to the timeout in milliseconds. [`WAITAOF`](https://docs.hotkv.com/commands/waitaof/index.md) waits for the writes to be in the append-only file on disk, on this node and on a number of replicas, and returns both counts: ``` $ printf "SET {cart1}:w 1\nWAIT 1 100\nWAIT 2 100\nWAITAOF 1 1 100\nWAITAOF 1 2 100\n" | redis-cli --no-raw -p 8100 OK (integer) 1 (integer) 1 1) (integer) 0 2) (integer) 0 1) (integer) 0 2) (integer) 0 ``` `WAIT 2` found one replica, because the shard has one. `WAITAOF` returned zeros here because the timeout of 100 ms ran out before the AOF reached the disk: with `appendfsync everysec` that takes up to a second. With a longer timeout, or with `WAITAOF 1 0 0`, which waits for the local AOF, both counts came back as 1: ``` 127.0.0.1:8100> SET {cart1}:t 2 OK 127.0.0.1:8100> WAITAOF 1 1 2000 1) (integer) 1 2) (integer) 1 ``` With the replica frozen, `WAIT` reports that the write did not reach it, and the write stays in place on the primary: ``` $ kill -STOP $ printf "SET {cart1}:w 2\nWAIT 1 300\nWAITAOF 1 1 300\n" | redis-cli --no-raw -p 8100 OK (integer) 0 1) (integer) 0 2) (integer) 0 $ kill -CONT ``` `WAIT` narrows the window in which a failover loses a write but does not close it (see [Where writes can be lost](#where-writes-can-be-lost)). ## Automatic failover When a primary stops answering, the cluster does this without an operator: 1. After `cluster-node-timeout` (default 15000 ms) of silence, each node flags the primary `PFAIL` (it shows as `fail?` in `CLUSTER NODES`). 2. When a majority of the primaries that serve slots report it, the node flags it `FAIL` and tells the others. 3. Its replica with the most data asks the primaries for votes. With a majority it becomes the primary, takes the slots and announces it. A shard with more than one replica ranks them by how much data each has. 4. The other nodes update their maps. Clients get `MOVED` to the new primary. The shard's other replicas follow the new primary, and the old primary rejoins as a replica of the new one when it comes back. A replica whose data is too old (older than `cluster-node-timeout` times `cluster-replica-validity-factor`, plus the replication heartbeat period, 10 seconds by default; `0` turns off this age check only), one that has never finished a sync, and one with `cluster-replica-no-failover yes` do not start an automatic failover. A shard without such a replica stays down until its primary returns or you promote a replica yourself. ### What it looks like The drill at the end of this page (see [the drill](#a-failover-drill)) kills a primary with `kill -9` while `writer.py` writes numbered keys, and prints a line each time the picture changes on a surviving node: the cluster state, the flags of the old primary and the role of its replica. With the default timeout the old primary goes from `master` to `master,fail?` (suspected) to `master,fail` (agreed), and the replica from `slave` to `master`. The shard is unavailable for about the node timeout plus a few seconds: the default 15 seconds of silence before the nodes suspect the primary, a few more for the majority to agree, and a fraction of a second for the election. The client gets `ConnectionError` for every write to that shard until the new primary takes over, then carries on. The drill's last line also shows the price of asynchronous replication: the writes the primary acknowledged but the replica had not received when it was killed are gone, and are counted as `missing`. The log of the replica that won, and of one of the nodes that voted: ``` --- log of 8103 (the replica) [hotkv] [info] Cluster: node 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it [hotkv] [warning] FAIL message received from 9f94ae9ebededa10d6b070be45745b51916ff313 (127.0.0.1:8104) about 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) [hotkv] [info] This is the best ranked replica and can initiate the election immediately. [hotkv] [info] Cluster: failover election for epoch 7: votes asked of the primaries [hotkv] [info] Cluster: Failover vote 1/3 (need 2) [hotkv] [info] Cluster: Failover vote 2/3 (need 2) [hotkv] [info] Cluster: this replica won the failover election: it takes over as the primary [hotkv] [info] Cluster: Completing failover [hotkv] [info] Replication: cluster failover - becoming primary (accepting replicas) --- log of 8101 (a voter) [hotkv] [info] Cluster: node 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) flagged PFAIL: a PING to it waited 15000 ms with nothing heard from it [hotkv] [warning] FAIL message received from 9f94ae9ebededa10d6b070be45745b51916ff313 (127.0.0.1:8104) about 30f273c04fb9eb0bd597d65985df7fc92ca1ffd3 (127.0.0.1:8102) [hotkv] [info] Cluster: voted for 4ccb024649e2904d33704bae0c3f3ba8fb149909 (127.0.0.1:8103) in the failover election of epoch 7 ``` With `--cluster-node-timeout 2000` the same sequence takes a little longer than the timeout instead of about 20 seconds. ### Choose the node timeout `cluster-node-timeout` is the length of silence after which a node counts another as failing, so it sets how long a real failure takes to notice and how patient the cluster is with a node that is only slow. A short timeout recovers faster, and also promotes a replica for a node that merely paused: an overloaded host, a stalled disk or a network blip longer than the timeout looks like a failure. Pick a value larger than the longest pause you see on your hosts and network. `CONFIG SET` changes it at runtime; set it to the same value on every node. ### Watching and restarting The old primary can be started again at any time with its own data directory and configuration. It finds out that another node took its slots, and becomes a replica of the new primary, then catches up: ``` $ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort 127.0.0.1:8100@18100 slave 4ae1e9d673d4d1f67fc15ac09987f85a96079ae7 connected 127.0.0.1:8101@18101 myself,master - connected 5461-10922 127.0.0.1:8102@18102 master - connected 10923-16383 127.0.0.1:8103@18103 slave f31014c5d868f136d477f68a908ae05a55800092 connected 127.0.0.1:8104@18104 master - connected 0-5460 127.0.0.1:8105@18105 slave 00cc163c39e228a537dbc06b5601bfa7f3cb781b connected ``` If the old primary comes back without its data (an empty directory), do not let it rejoin with the old ID before you check the roles: see [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md). ## Network partitions A node cut off from most of the cluster must stop accepting writes, because the majority will elect a new primary for its slots. HotKV does this: a node that cannot reach a majority of the primaries marks the cluster as down and refuses keys. The example uses two hosts: five nodes on `10.0.0.1` (primaries 8500 and 8501, replicas 8503, 8504 and 8505) and one on `10.0.0.2` (the primary 8502, whose replica 8505 is on the other host), all with `cluster-node-timeout 5000`: ``` $ hotkv --port 8500 --bind 10.0.0.1 --cluster-enabled yes --protected-mode no --dir node-8500 --daemonize yes --logfile hotkv.log --cluster-node-timeout 5000 --appendonly yes ``` (Start the nodes on `10.0.0.1` this way on ports 8500, 8501, 8503, 8504 and 8505, and the one on `10.0.0.2` with `--bind 10.0.0.2` on port 8502.) To simulate a partition, cut the nodes on `10.0.0.1` off from `10.0.0.2` on the client and bus ports by dropping the packets it sends: ``` $ sudo iptables -I INPUT -p tcp -s 10.0.0.2 -m multiport --dports 8500:8599,18500:18599 -j DROP $ sudo iptables -I INPUT -p tcp -s 10.0.0.2 -m multiport --sports 8500:8599,18500:18599 -j DROP ``` (Remove the rules afterwards with the same commands and `-D` in place of `-I`.) Suppose a client writes `{session1}` keys straight to the isolated primary on `10.0.0.2` throughout. After 5 seconds the nodes on the isolated side flag the others `PFAIL`, and then: ``` --- CLUSTER NODES seen from the majority (10.0.0.1) 10.0.0.2:8502@18502 master,fail - disconnected 10.0.0.1:8500@18500 myself,master - connected 0-5460 10.0.0.1:8501@18501 master - connected 10923-16383 10.0.0.1:8503@18503 slave c88860a72dce6c55c08f09749533856e7dca2c27 connected 10.0.0.1:8504@18504 slave d8e3515ab2d8888ce9595e5f4099cc3b0996f2a5 connected 10.0.0.1:8505@18505 master - connected 5461-10922 --- CLUSTER NODES seen from the minority node (10.0.0.2) 10.0.0.2:8502@18502 myself,master - connected 5461-10922 10.0.0.1:8500@18500 master,fail? - disconnected 0-5460 10.0.0.1:8501@18501 master,fail? - disconnected 10923-16383 10.0.0.1:8503@18503 slave,fail? c88860a72dce6c55c08f09749533856e7dca2c27 disconnected 10.0.0.1:8504@18504 slave,fail? d8e3515ab2d8888ce9595e5f4099cc3b0996f2a5 disconnected 10.0.0.1:8505@18505 slave,fail? c78599fe62b4ee129b6e93e3032646aea0fce49e disconnected ``` The majority side promotes the replica 8505 and keeps serving. The minority node logs that it is alone, and from then on answers `CLUSTERDOWN`: ``` Cluster: the cluster is down: this node is in a minority partition, reaching no majority of the primaries that serve slots ``` The isolated primary keeps accepting writes for a while after the cut, because it needs the node timeout to notice. Those writes go nowhere: after the network is restored, the old primary becomes a replica of 8505 and the writes it acknowledged during the window are gone. A lower `cluster-node-timeout` shortens this window and a higher one stretches it. The rule is the same as for a failure: only the side with a majority of the primaries keeps serving, so run at least three primaries and spread them over hosts or zones so that no single host holds a majority. ## Start a failover yourself [`CLUSTER FAILOVER`](https://docs.hotkv.com/commands/cluster-failover/index.md) on a replica promotes it. Use it to move a primary to another host before maintenance, or to recover when automatic failover cannot run. There are three modes. | Mode | What it does | Use it when | | --- | --- | --- | | `CLUSTER FAILOVER` | The primary pauses writes, the replica catches up with every write, then wins an election. No acknowledged write is lost. | The primary is up and you choose the moment. | | `CLUSTER FAILOVER FORCE` | Skips the handshake with the primary and runs the election straight away. A majority of the primaries must still vote for it. | The primary is down or stuck but the cluster has a majority. | | `CLUSTER FAILOVER TAKEOVER` | Takes over at once with a new epoch, without an election. | A majority of the primaries cannot be reached. | Every mode must be sent to a replica; sent to a primary it replies `ERR You should send CLUSTER FAILOVER to a replica`. Start one failover at a time and wait until the replica has become a primary (`ROLE`, or `CLUSTER NODES`) before you start the next. Elections that begin at the same moment in different shards can collide on the same epoch, and the replica that loses never takes over, although `CLUSTER FAILOVER` answered `OK`: its log says `Manual failover timed out. It did not win an election in time`. If that happens, check the roles and send the command again. ### The default mode The default mode needs the primary to answer. It pauses the writes of its slots for a moment, so clients wait instead of failing, and no acknowledged write is lost: ``` 127.0.0.1:8103> CLUSTER FAILOVER OK $ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort 127.0.0.1:8100@18100 slave 3bfee53e12421ac6c914bf03042dc2a2fde68fb4 connected 127.0.0.1:8101@18101 myself,master - connected 5461-10922 127.0.0.1:8102@18102 master - connected 10923-16383 127.0.0.1:8103@18103 master - connected 0-5460 127.0.0.1:8104@18104 slave 147c5200dfb75e7fcbe095c22dc93cc590cb7cac connected 127.0.0.1:8105@18105 slave f90cb48df290ae2ef2328925012abcebd48204e1 connected ``` The replica logs that the primary was paused, then that it caught up with the primary and is starting the election, and wins it. The default mode has `cluster-manual-failover-timeout` (default 5000 ms) to finish. If the primary does not answer, it gives up and logs it, and the replica stays a replica. ### FORCE With the primary frozen (`kill -STOP`), the default mode fails, and `FORCE` works: ``` $ kill -STOP 127.0.0.1:8104> CLUSTER FAILOVER OK $ redis-cli -p 8104 ROLE | head -n 1 slave Manual failover timed out. It did not catch up with primary 216cb28fed6439091f6ca74f31c6a4a54bb7e7ed in time (CLUSTER FAILOVER FORCE or TAKEOVER override this) 127.0.0.1:8104> CLUSTER FAILOVER FORCE OK $ redis-cli -p 8104 ROLE | head -n 1 master $ kill -CONT ``` The first reply is `OK` because the failover started, and the `ROLE` after it still says `slave` once the timeout has run out. The log line is what the replica writes when the timeout runs out. After `FORCE` the replica becomes the primary within a couple of seconds. When the frozen primary is resumed it finds it has lost its slots and turns into a replica. Writes it accepted after the last one the replica had are lost. ### TAKEOVER When most primaries are unreachable, nobody can vote, and `FORCE` runs out of time. `TAKEOVER` does not need a vote. Here two of the three primaries are frozen: ``` $ kill -STOP 127.0.0.1:8104> CLUSTER FAILOVER FORCE OK $ redis-cli -p 8104 ROLE | head -n 1 slave Manual failover timed out. It did not win an election in time 127.0.0.1:8104> CLUSTER FAILOVER TAKEOVER OK $ redis-cli -p 8104 ROLE | head -n 1 master $ kill -CONT ``` Use `TAKEOVER` only when you know the old primary will not be reached by clients that keep writing to it: it takes the slots without asking the cluster. When the other nodes come back, the node with the newer epoch wins and the old primary becomes its replica, losing what it had that the replica did not. A replica that never finished its first sync serves the shard with incomplete data after a takeover, so check it first. `CLUSTER FAILOVER` with an unknown word fails with `syntax error`. ## Planned restarts To restart or upgrade a primary without a pause for its clients, hand its slots to a replica first and then stop it. The commands do it in one step. `SHUTDOWN SAFE` refuses on a primary that serves slots, because its slots would be down until a replica took them: ``` 127.0.0.1:8100> SHUTDOWN SAFE (error) ERR Errors trying to SHUTDOWN. Check logs. SHUTDOWN SAFE: this primary serves slots, which would be down until a replica took them over; not shutting down ``` `SHUTDOWN FAILOVER` hands the slots to a replica that has every write, and then stops. Clients see no error: ``` 127.0.0.1:8100> SHUTDOWN FAILOVER $ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort 127.0.0.1:8100@18100 master - disconnected 127.0.0.1:8101@18101 myself,master - connected 5461-10922 127.0.0.1:8102@18102 master - connected 10923-16383 127.0.0.1:8103@18103 master - connected 0-5460 127.0.0.1:8104@18104 slave 791b47c22caa971b3ae8cef340313db0480b135c connected 127.0.0.1:8105@18105 slave 24f55d649c24000381721f4691bd432915b9bfe2 connected ``` (The old primary shows as `disconnected` for a moment, until the others drop its entry as it is now the replica of 8103.) Start the node again and it comes back as a replica. `SHUTDOWN SAFE` on a replica, or on a primary without slots, just stops it. `SHUTDOWN FAILOVER` outside cluster mode is refused with `ERR SHUTDOWN FAILOVER is only supported in cluster mode.` To get the same from `systemctl stop` or a `SIGTERM`, set `shutdown-on-sigterm failover` (and `shutdown-on-sigint` for `SIGINT`): ``` $ redis-cli -p 8100 CONFIG GET shutdown-on-sigterm shutdown-on-sigterm failover $ kill -TERM ``` If no replica is ready, a primary logs `Unable to find a replica to perform the auto failover on shutdown.` and stops; the cluster then promotes a replica as for any other failure. ## Replica migration When a primary loses its last replica, a spare replica of another shard can move to it, so a second failure does not take the shard down. It is on by default (`cluster-allow-replica-migration yes`), and a shard only lends a replica while it keeps more than `cluster-migration-barrier` (default `1`) of them. Take a cluster of three primaries with two replicas each (nodes 8100 to 8108). When both replicas of the primary on port 8101, nodes 8107 and 8108, are killed with `kill -9`, the `CLUSTER NODES` listing a few seconds later shows a replica of another shard moving over: ``` $ redis-cli -p 8100 CONFIG GET cluster-migration-barrier cluster-migration-barrier 1 $ redis-cli -p 8100 CLUSTER NODES | awk "{print \$2, \$3, \$4}" | cut -d@ -f2- | sort 18100 myself,master - 18101 master - 18102 master - 18103 slave 58f6d7f3fce5bdee819d27a880087133755fac78 18104 slave 58f6d7f3fce5bdee819d27a880087133755fac78 18105 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9 18106 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9 18107 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52 18108 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52 $ redis-cli -p 8100 CLUSTER NODES | awk "{print \$2, \$3, \$4}" | cut -d@ -f2- | sort 18100 myself,master - 18101 master - 18102 master - 18103 slave 58f6d7f3fce5bdee819d27a880087133755fac78 18104 slave 58f6d7f3fce5bdee819d27a880087133755fac78 18105 slave 487a60dfb7ae5e1379913a57adfbdbba95a72e52 18106 slave 7d5d376e0b2691b40c2b3e665fbabdc3cf4125b9 18107 slave,fail 487a60dfb7ae5e1379913a57adfbdbba95a72e52 18108 slave,fail 487a60dfb7ae5e1379913a57adfbdbba95a72e52 Migrating to orphaned primary: replica 352ae1618d3f148c8e31ef6a44582b8803887af8 moves to 487a60dfb7ae5e1379913a57adfbdbba95a72e52 (127.0.0.1:8101) of shard fcb3d0a02 ``` Node 8105, which followed another primary, now follows the primary of 8101. The last line is from the log of node 8105. The move waits a few seconds after the loss, so that the shard's own replicas have time to report. Set `cluster-allow-replica-migration no` if you place replicas by hand and do not want them to move. ## Keep a replica from taking over A replica with `cluster-replica-no-failover yes` shows the flag `nofailover` in `CLUSTER NODES`, is left out of the ranking and never starts an automatic failover. Use it for a replica you keep for backups or reads on a weaker host. `CLUSTER FAILOVER` still promotes it when you ask. When it was the only replica and the primary died, nothing promoted it until `CLUSTER FAILOVER FORCE`: ``` 127.0.0.1:8104> CONFIG SET cluster-replica-no-failover yes OK $ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3 | grep slave 127.0.0.1:8104@18104 slave,nofailover 127.0.0.1:8105@18105 slave 127.0.0.1:8103@18103 slave $ kill -9 $ redis-cli -p 8101 CLUSTER NODES | cut -d" " -f2,3,4,9 | sort 127.0.0.1:8100@18100 master,fail - 0-5460 ... 127.0.0.1:8104@18104 slave,nofailover e25c9fc75c319b0b26c52a7c3f461e7ecd74bd01 ... $ redis-cli -p 8101 CLUSTER INFO | grep cluster_state cluster_state:fail 127.0.0.1:8104> CLUSTER FAILOVER FORCE OK $ redis-cli -p 8101 CLUSTER INFO | grep cluster_state cluster_state:ok ``` ## Where writes can be lost Replication is asynchronous, so a write the primary acknowledged can be lost in the cases below. The list is about replication and failover. It leaves out what any server does: eviction and expiry remove keys, and a node that crashes and restarts from its own files loses the writes its append-only file had not flushed yet (about a second with `appendfsync everysec`). The cases are: - **The primary fails before its replica has the write.** A failover promotes the replica, which does not have it. Only the few writes in flight at that moment are affected. `WAIT 1 timeout` after a write makes the client wait until a replica has it, and shrinks the window to the time between the write and the `WAIT` reply; a client that did not get the reply must treat the write as unknown. - **A primary is cut off from the majority.** It keeps acknowledging writes until it notices, up to about the node timeout, and loses them when it rejoins as a replica (see [Network partitions](#network-partitions)). - **`FORCE` or `TAKEOVER` on a replica that is behind**, or a primary that was frozen and keeps its late writes. - **A restart without data.** A primary that comes back with an empty directory has nothing to give, and its replicas copy the emptiness. Check the roles before you start a node that lost its disk. The planned ways of moving a primary (`CLUSTER FAILOVER`, `SHUTDOWN FAILOVER`) lose no acknowledged write: the primary stops accepting writes, the replica catches up, and only then does it take over. ## A failover drill Run this on a practice cluster that lives on one host, such as the one from the [quick start](https://docs.hotkv.com/clustering/quick-start/index.md), before you trust a cluster in production. It needs `writer.py` from [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md#check-that-no-write-is-lost) in the current directory and the `redis` package (install it in a virtual environment, as shown in [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md#check-that-no-write-is-lost)). It picks a primary that has a replica, kills it, and prints a timeline of the failover. ```python """Kill one primary of a practice cluster on this host while writer.py runs, and report what happened. Usage: python3 drill.py HOST:PORT (writer.py must be in the current directory) """ import os import signal import subprocess import sys import time import redis def connect(addr): host, port = addr.rsplit(":", 1) return redis.Redis(host=host, port=int(port), decode_responses=True) def nodes(client): result = [] for line in client.execute_command("CLUSTER", "NODES").splitlines(): f = line.split() result.append({"id": f[0], "addr": f[1].split("@")[0], "flags": f[2].replace("myself,", ""), "master": f[3], "slots": f[8:]}) return result seed = connect(sys.argv[1]) cluster = nodes(seed) victim = next(n for n in cluster if n["flags"] == "master" and n["slots"] and any(r["master"] == n["id"] for r in cluster)) replica = next(r for r in cluster if r["master"] == victim["id"]) survivor = next(n for n in cluster if n["flags"] == "master" and n["slots"] and n is not victim) pid = connect(victim["addr"]).info("server")["process_id"] writer = subprocess.Popen([sys.executable, "writer.py", sys.argv[1], "60"], stdout=open("drill-writer.out", "w"), stderr=subprocess.STDOUT) time.sleep(5) print(f"primary {victim['addr']} (pid {pid}), replica {replica['addr']}") start = time.time() os.kill(pid, signal.SIGKILL) print(f"{0:5.1f}s kill -9 {pid}") watch, last = connect(survivor["addr"]), None while time.time() - start < 120: try: state = watch.execute_command("CLUSTER", "INFO").split("cluster_state:")[1].split()[0] flags = {n["addr"]: n["flags"] for n in nodes(watch)} view = (state, flags[victim["addr"]], flags[replica["addr"]]) except redis.RedisError: time.sleep(0.1) continue if view != last: print(f"{time.time() - start:5.1f}s cluster_state={view[0]} " f"old primary: {view[1]} replica: {view[2]}") last = view if view[0] == "ok" and view[2] == "master": break time.sleep(0.1) writer.wait() print(open("drill-writer.out").read().strip().splitlines()[-1]) ``` Read the last line: `errors` and `longest_stall` are what your clients lived through, and `missing` is the number of acknowledged writes that were lost. Start the killed node again afterwards (it comes back as a replica), and repeat the drill with your real client library in place of `writer.py` to see what your application does. ## Next steps - [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md) for the alerts to set on failover and replication lag. - [Upgrades](https://docs.hotkv.com/clustering/upgrades/index.md) for the planned way to replace every node. - [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md) to add replicas and move slots. - [Replication](https://docs.hotkv.com/operate/replication/index.md) for replication outside cluster mode.