# Cluster backup and restore > Back up a HotKV cluster node by node, restore a replica, a primary or the whole cluster, recover from a lost nodes.conf, and move a cluster to new addresses. URL: https://docs.hotkv.com/clustering/backup-and-restore/ **Availability:** Not on HotKV Cloud. On-Premise: no license needed. A cluster has no single backup file. Each node holds its own part of the data and its own record of the cluster, so a backup is a set of files, one set per node. This page shows what to copy, how to take a backup that restores cleanly, and how to bring back a node, a shard or the whole cluster. Every procedure ends with a check, and the examples use a cluster on one machine: three primaries and three replicas on ports 7500 to 7505, with a data directory per node under `/var/lib/hotkv`, started with `--appendonly yes`. The commands run from `/var/lib/hotkv`, where each node has a directory named after its port, and "start the node" means this, with the port as the argument (`start 7503`). Define it once in your shell. Besides the settings above it writes a PID file, which the transcripts use to stop a node: ```bash start() { hotkv --port $1 --bind ${2:-127.0.0.1} --cluster-enabled yes --dir "$PWD/${3:-$1}" \n --pidfile "$PWD/${3:-$1}/hotkv.pid" --logfile hotkv.log --daemonize yes \n --appendonly yes --cluster-node-timeout 5000 } ``` The second and third arguments are used only when a node moves to a new address and directory (see [Move the cluster to new addresses](#move-the-cluster-to-new-addresses)). ## What a node's state is | File (default name) | Setting | What it holds | | --- | --- | --- | | `dump.hkv` | `dbfilename` | A snapshot of the node's data. It appears after a `SAVE`, a `BGSAVE` or a scheduled save. | | `appendonly.hkv` | `appendfilename` | The append-only file (AOF), when `appendonly yes` is set. | | `nodes.conf` | `cluster-config-file` | The node's ID and everything it knows about the cluster: other nodes, slot owners, epochs. | All three live in the node's `dir`. You can read the names from a running node: ```bash redis-cli -p 7500 CONFIG GET dbfilename redis-cli -p 7500 CONFIG GET appendfilename redis-cli -p 7500 CONFIG GET cluster-config-file ``` ```text dbfilename dump.hkv appendfilename appendonly.hkv cluster-config-file nodes.conf ``` Two more things are per node and are not in the data files: the license file (`license-path`), and the key of encryption at rest (`encryption.key-file`, or your key service). Back them up with your configuration management. A restored data file that was written with encryption at rest cannot be read without its key. `nodes.conf` matters as much as the data. It carries the node's identity, so a node restored without it is a new node (see [A node loses nodes.conf](#a-node-loses-nodesconf)). Keep it with the data files, taken at the same time. ## Persistence settings first A backup can only contain what a node wrote down. Decide how much you can lose: - `appendonly yes` with `appendfsync everysec` (the default) writes every change to the AOF and flushes it once a second, so a crash of the host loses about a second. `appendfsync always` flushes every write, and `no` leaves it to the operating system. - Snapshots (`save` points, `BGSAVE`) are what you copy for backups. A node with `appendonly no` and no save points keeps data only in memory. - `WAITAOF numlocal numreplicas timeout` makes a client wait until its write is on disk locally and on that many replicas. On a primary with `appendonly yes` and one replica that also has it on: ```bash redis-cli -p 7500 WAITAOF 1 1 2000 ``` ```text 1 1 ``` Use `WAITAOF` for the writes you must not lose, and `WAIT` when replica copies are enough. The data on a replica is a copy of its primary's, so a replica can also be the node you back up. ## Take a backup Ask every node for a snapshot, wait until each has finished, then copy the files. Run the commands from `/var/lib/hotkv`, where the node directories are: ```bash for port in 7500 7501 7502 7503 7504 7505; do redis-cli -p $port BGSAVE; done ``` ```text Background saving started Background saving started Background saving started Background saving started Background saving started Background saving started ``` ```bash for port in 7500 7501 7502 7503 7504 7505; do redis-cli -p $port INFO persistence | grep -E 'rdb_bgsave_in_progress|rdb_last_bgsave_status' | tr '\n' ' '; echo done ``` ```text rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok ``` A snapshot is finished when `rdb_bgsave_in_progress` is `0` and `rdb_last_bgsave_status` is `ok`. Copy `dump.hkv`, `nodes.conf` and, when you run the AOF, `appendonly.hkv`: ```bash mkdir backup for port in 7500 7501 7502 7503 7504 7505; do mkdir backup/$port cp $port/dump.hkv $port/appendonly.hkv $port/nodes.conf backup/$port/ done ls backup/7500 ``` ```text appendonly.hkv dump.hkv nodes.conf ``` Then move `backup` off the machine. Each node's files are consistent with that node, but the files of different nodes are taken a moment apart, so writes made during the backup can be in one shard's files and not another's. If you need the same moment everywhere, stop your writers while the snapshots run. **Which file a node loads.** A node started with `appendonly yes` that has an AOF loads the AOF, even when a snapshot is older or newer. A node that has only a snapshot loads the snapshot, whether `appendonly` is `yes` or `no`. So to restore from a snapshot alone, remove the node's old `appendonly.hkv` first, or restore the AOF from the same backup. To spare the primaries, take the files of a replica instead: it holds the same data as its primary. Keep the `nodes.conf` of every node you will restore, including the primaries'. > **Caution** > The AOF grows while the node runs, so a copy of it can end in the middle of a > write. HotKV loads such a file without the unfinished last write, but a file > damaged anywhere else stops the start. The snapshot you just took is complete, > which makes it the safest file to rely on. ## Restore a replica A replica's data is a copy, so the simplest restore is to let it copy again. **It lost its data files but kept `nodes.conf`.** The node comes back with its identity and copies the data from its primary: ```bash kill $(cat 7503/hotkv.pid); sleep 2 rm 7503/dump.hkv 7503/appendonly.hkv ``` Start the node with `start 7503`, then wait for the link and check the size: ```bash until redis-cli -p 7503 INFO replication | grep -q master_link_status:up; do sleep 1; done; redis-cli -p 7503 INFO replication | grep -E "role|master_link_status"; redis-cli -p 7503 DBSIZE ``` ```text role:slave master_link_status:up 108 ``` 108 was the number of keys on its primary. **Check:** `DBSIZE` on the replica equals `DBSIZE` on its primary, and `redis-cli --cluster check` reports no errors. **It lost everything, `nodes.conf` too.** It starts as a new, empty node with a new ID, unknown to the cluster: ```bash kill $(cat 7503/hotkv.pid); sleep 2 rm -r 7503; mkdir 7503 ``` Start it with `start 7503`, and it knows only itself, while the cluster still lists the old ID, now failing and without an address: ```text $ redis-cli -p 7503 CLUSTER NODES cbd5cf8775d821dd80deec411712e6f91364b66b 127.0.0.1:7503@17503 myself,master - 0 0 0 connected $ redis-cli -p 7500 CLUSTER NODES | cut -c1-120 a5a0d70bfb22f7a5272d477b8b644a573b5a3c5c 127.0.0.1:7500@17500 myself,master - 0 0 1 connected 0-5460 2fdb058c1e973de99069115a3fd996755c1ece38 127.0.0.1:7501@17501 master - 0 1791068463156 2 connected 5461-10922 dd2b2ca966953a9bf39e379db924d0f8575a6129 127.0.0.1:7502@17502 master - 0 1791068465159 3 connected 10923-16383 f2f83c7b5907e4f1b9265cfbac6c744cd2176234 :0@0 slave,fail,noaddr 2fdb058c1e973de99069115a3fd996755c1ece38 1791068464157 1 ... ``` (Shortly after the old node stops, its entry may still show its address and the flag `slave` alone.) Introduce the new node to the cluster and make it a replica of the primary the old one followed: the old entry's fourth field, once the cluster has flagged it `slave,fail`. The new node copies the data from there: ```bash redis-cli -p 7500 CLUSTER MEET 127.0.0.1 7503 sleep 3 OLDPRIMARY=$(redis-cli -p 7500 CLUSTER NODES | grep 'slave,fail' | cut -d' ' -f4) redis-cli -p 7503 CLUSTER REPLICATE $OLDPRIMARY sleep 6 redis-cli -p 7503 INFO replication | grep -E "role|master_link_status" ``` ```text OK OK role:slave master_link_status:up ``` The old entry is still listed: ```bash redis-cli -p 7500 CLUSTER NODES | grep noaddr ``` ```text f2f83c7b5907e4f1b9265cfbac6c744cd2176234 :0@0 slave,fail,noaddr 2fdb058c1e973de99069115a3fd996755c1ece38 1791068464157 1791068459150 4 disconnected ``` Remove it by telling every node to forget it: ```bash OLD=$(redis-cli -p 7500 CLUSTER NODES | grep noaddr | cut -d' ' -f1) for port in 7500 7501 7502 7503 7504 7505; do redis-cli -p $port CLUSTER FORGET $OLD done ``` ```text OK OK OK ERR Unknown node f2f83c7b5907e4f1b9265cfbac6c744cd2176234 OK OK ``` The node that never knew the old ID answers `ERR Unknown node ...`, which you can ignore (which node says so differs, since the new node may or may not have heard of the old one). Forgotten nodes stay forgotten for `cluster-blacklist-ttl` seconds (60 by default), so run the loop on all nodes close together. The old entry can also disappear on its own once the new node has joined, so always look before you forget. **Check:** ```bash redis-cli -p 7500 CLUSTER NODES | grep -c fail redis-cli --cluster check 127.0.0.1:7500 | tail -n 3 ``` ```text 0 >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. ``` There are no `fail` entries, the new node is a `slave` of the right primary, and the check ends with `[OK] All 16384 slots covered.` ## Replace a primary Let its replica take over first. A stopped primary is replaced automatically by its replica (see [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md)), or run `CLUSTER FAILOVER` on the replica to do it on purpose. Once the old primary is only a replica, or gone, treat it as a lost replica: restore it as shown above, or re-create it with the steps for [a node that loses nodes.conf](#a-node-loses-nodesconf). ## Restore the whole cluster in place To roll the entire cluster back to a backup, stop every node, put each node's files back, and start them. The cluster comes back as it was at the backup, with the same node IDs. First, write some data after the backup so that the restore is visible (these keys are not in the backup): ```bash redis-cli --cluster check 127.0.0.1:7500 | head -n 4 ``` ```text 127.0.0.1:7500 (bbfb68f7...) -> 113 keys | 5461 slots | 1 slaves. 127.0.0.1:7502 (98295e66...) -> 126 keys | 5461 slots | 1 slaves. 127.0.0.1:7501 (6974d3c3...) -> 111 keys | 5462 slots | 1 slaves. [OK] 350 keys in 3 masters. ``` ```bash for port in 7500 7501 7502 7503 7504 7505; do kill $(cat $port/hotkv.pid); done sleep 3 for port in 7500 7501 7502 7503 7504 7505; do rm $port/dump.hkv $port/appendonly.hkv $port/nodes.conf cp backup/$port/* $port/ done ``` Start the six nodes with `start`, one call for each port. Then wait until the cluster is ready and compare: ```bash until redis-cli -p 7500 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done; redis-cli --cluster check 127.0.0.1:7500 | head -n 4 ``` ```text 127.0.0.1:7500 (bbfb68f7...) -> 100 keys | 5461 slots | 1 slaves. 127.0.0.1:7501 (6974d3c3...) -> 92 keys | 5462 slots | 1 slaves. 127.0.0.1:7502 (98295e66...) -> 108 keys | 5461 slots | 1 slaves. [OK] 300 keys in 3 masters. ``` ```bash redis-cli -c -p 7500 GET key:300 redis-cli -c -p 7500 GET key:350 redis-cli -p 7503 INFO replication | grep -E "role|master_link_status" ``` ```text value:300 (nil) role:slave master_link_status:up ``` The 300 keys of the backup are back, `key:350` (written afterwards) is gone, and the replicas are connected (`redis-cli` prints the `(nil)` as an empty line when its output is not a terminal). **Check:** the key count matches the backup, a sample of keys reads back, `CLUSTER INFO` says `cluster_state:ok`, and every replica shows `master_link_status:up`. Restore the files of a shard's primary and replicas from the same backup, and the shards of one cluster from the same backup. To bring back one node of a running cluster, restore it from its replica or primary as shown above. ## A node loses nodes.conf If a node's data is intact and only `nodes.conf` is gone, it starts as a new node with a new ID and loads its old data. Here, primary 7500 lost the file while its replica took over its slots: ```bash kill $(cat 7500/hotkv.pid); sleep 2 rm 7500/nodes.conf ``` After `start 7500`, the node knows only itself, holds keys and owns no slots, while the cluster lists its old ID as failed: ```text $ redis-cli -p 7501 CLUSTER NODES | cut -c1-110 cbd5cf8775d821dd80deec411712e6f91364b66b 127.0.0.1:7503@17503 slave 2fdb058c1e973de99069115a3fd996755c1ece38 0 2fdb058c1e973de99069115a3fd996755c1ece38 127.0.0.1:7501@17501 myself,master - 0 0 2 connected 5461-10922 a5a0d70bfb22f7a5272d477b8b644a573b5a3c5c 127.0.0.1:7500@17500 master,fail - 1791068481194 1791068476688 1 disc 2dceabab81d7383df895ef008ee0e605fa3a448d 127.0.0.1:7504@17504 slave dd2b2ca966953a9bf39e379db924d0f8575a6129 0 af4718e90d191585d3f0d4f0eec153e26e1c845c 127.0.0.1:7505@17505 master - 0 1791068497251 7 connected 0-5460 dd2b2ca966953a9bf39e379db924d0f8575a6129 127.0.0.1:7502@17502 master - 0 1791068501256 3 connected 10923-16383 $ redis-cli -p 7500 CLUSTER NODES f3ae190d891c5b743fd6f717532bed475c7f5b50 127.0.0.1:7500@17500 myself,master - 0 0 0 connected $ redis-cli -p 7500 DBSIZE 100 ``` The data on the new node is stale: its old shard has a new primary, 7505. The node cannot become a replica while it holds that data: ```bash NEW=$(redis-cli -p 7501 CLUSTER NODES | grep ' master ' | grep 0-5460 | cut -d' ' -f1) redis-cli -p 7501 CLUSTER MEET 127.0.0.1 7500 sleep 3 redis-cli -p 7500 CLUSTER REPLICATE $NEW redis-cli -p 7500 FLUSHALL redis-cli -p 7500 CLUSTER REPLICATE $NEW ``` ```text OK ERR To set a master the node must be empty and without assigned slots. OK OK ``` After `FLUSHALL` the replicate works, and the node copies the shard's current data from the new primary. Then forget the old ID on every node, as above (here the failed entry is the one with `noaddr`, and the new node, which never knew it, answers `ERR Unknown node`). **Check:** ```bash redis-cli -p 7500 INFO replication | grep -E "role|master_link_status" redis-cli -p 7500 DBSIZE redis-cli --cluster check 127.0.0.1:7501 | tail -n 4 ``` ```text role:slave master_link_status:up 100 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. ``` If you lose `nodes.conf` from every node, restore them from your backup. A cluster whose `nodes.conf` files are all gone has to be created again with `redis-cli --cluster create` on empty nodes, and its data loaded from your snapshots or with a migration tool. ## Move the cluster to new addresses To move a cluster to new hosts, or to change the addresses of its nodes, stop every node, copy each node's directory to its new place, and fix the addresses. `nodes.conf` records every node's address, and a node derives its own from its settings but learns the others from the file. The example moves a cluster from `127.0.0.1` to `127.0.0.2` (a different address on the same machine). **Edit `nodes.conf`.** Take a fresh snapshot, stop the nodes, copy, and replace the old address with the new one in each copied `nodes.conf`: ```bash for port in 7500 7501 7502 7503 7504 7505; do redis-cli -p $port BGSAVE done sleep 3 for port in 7500 7501 7502 7503 7504 7505; do kill $(cat $port/hotkv.pid); done sleep 3 mkdir new for port in 7500 7501 7502 7503 7504 7505; do mkdir new/$port cp $port/dump.hkv $port/appendonly.hkv $port/nodes.conf new/$port/ sed -i 's/127\.0\.0\.1/127.0.0.2/g' new/$port/nodes.conf done grep -c 127.0.0.2 new/7500/nodes.conf ``` ```text Background saving started Background saving started Background saving started Background saving started Background saving started Background saving started 6 ``` Start the nodes from the new directories, with their new `bind` (and any new announce settings), then check. With the function above, that is `start $port 127.0.0.2 new/$port` for each port: ```bash redis-cli -h 127.0.0.2 -p 7500 CLUSTER INFO | head -n 2 redis-cli --cluster check 127.0.0.2:7500 | head -n 4 redis-cli -c -h 127.0.0.2 -p 7500 GET key:123 ``` ```text cluster_enabled:1 cluster_state:ok 127.0.0.2:7501 (2fdb058c...) -> 92 keys | 5462 slots | 1 slaves. 127.0.0.2:7502 (dd2b2ca9...) -> 108 keys | 5461 slots | 1 slaves. 127.0.0.2:7505 (af4718e9...) -> 100 keys | 5461 slots | 1 slaves. [OK] 300 keys in 3 masters. value:123 ``` **Or introduce the new addresses.** Without editing the files, the nodes start with the new bind address but still hold the old addresses of their peers: `CLUSTER NODES` shows the others as `disconnected` at their old addresses. This example uses `127.0.0.3` as the new address. Tell one node about the others with `CLUSTER MEET` at their new addresses: ```bash for port in 7501 7502 7503 7504 7505; do redis-cli -h 127.0.0.3 -p 7500 CLUSTER MEET 127.0.0.3 $port done ``` ```text OK OK OK OK OK ``` The other nodes are now known at their new addresses, and `redis-cli --cluster check` run against the first node looks healthy. But the node you ran the `MEET` commands on is still known to every other node at its old address, so a client that is sent to it fails: ```text $ redis-cli -h 127.0.0.3 -p 7501 CLUSTER NODES | grep ":7500@" | cut -c1-110 6833e519e8bf0a3cab238b594189bb7bae1f52a0 127.0.0.1:7500@17500 master - 1791068911554 1791068911432 1 disconnec $ redis-cli -c -h 127.0.0.3 -p 7501 GET key:123 Could not connect to Redis at 127.0.0.1:7500: Connection refused ``` Introduce that node from one of the others as well: ```bash redis-cli -h 127.0.0.3 -p 7501 CLUSTER MEET 127.0.0.3 7500 sleep 8 redis-cli -h 127.0.0.3 -p 7501 CLUSTER NODES | grep ":7500@" | cut -c1-110 redis-cli -c -h 127.0.0.3 -p 7501 GET key:123 redis-cli -c -h 127.0.0.3 -p 7502 GET key:1 ``` ```text OK 6833e519e8bf0a3cab238b594189bb7bae1f52a0 127.0.0.3:7500@17500 master - 0 1791068938505 1 connected 0-5460 value:123 value:1 ``` After that, the node lists itself at the new address too. Editing `nodes.conf` is the cleaner way, because the cluster is whole as soon as the nodes start, and nothing is left at an old address. Whichever you use, update your clients and firewalls with the new addresses, and run the checks: the key count, a sample of keys, and `CLUSTER INFO`. ## See also - [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md) - [Scaling](https://docs.hotkv.com/clustering/scaling/index.md): adding and removing nodes without a restore. - [Persistence](https://docs.hotkv.com/persistence/index.md): snapshots and the AOF in detail. - [Production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md)