Cluster backup and restore

Last updated 12 Oct 2026

Not on HotKV CloudOn-Premise: no license needed

A cluster has no single backup file. Each node holds its own part of the data and its own record of the cluster, so a backup is a set of files, one set per node. This page shows what to copy, how to take a backup that restores cleanly, and how to bring back a node, a shard or the whole cluster. Every procedure ends with a check, and the examples use a cluster on one machine: three primaries and three replicas on ports 7500 to 7505, with a data directory per node under /var/lib/hotkv, started with --appendonly yes.

The commands run from /var/lib/hotkv, where each node has a directory named after its port, and “start the node” means this, with the port as the argument (start 7503). Define it once in your shell. Besides the settings above it writes a PID file, which the transcripts use to stop a node:

start() {
  hotkv --port $1 --bind ${2:-127.0.0.1} --cluster-enabled yes --dir "$PWD/${3:-$1}" \n    --pidfile "$PWD/${3:-$1}/hotkv.pid" --logfile hotkv.log --daemonize yes \n    --appendonly yes --cluster-node-timeout 5000
}

The second and third arguments are used only when a node moves to a new address and directory (see Move the cluster to new addresses).

What a node’s state is

File (default name)	Setting	What it holds
`dump.hkv`	`dbfilename`	A snapshot of the node’s data. It appears after a `SAVE`, a `BGSAVE` or a scheduled save.
`appendonly.hkv`	`appendfilename`	The append-only file (AOF), when `appendonly yes` is set.
`nodes.conf`	`cluster-config-file`	The node’s ID and everything it knows about the cluster: other nodes, slot owners, epochs.

All three live in the node’s dir. You can read the names from a running node:

redis-cli -p 7500 CONFIG GET dbfilename
redis-cli -p 7500 CONFIG GET appendfilename
redis-cli -p 7500 CONFIG GET cluster-config-file

dbfilename
dump.hkv
appendfilename
appendonly.hkv
cluster-config-file
nodes.conf

Two more things are per node and are not in the data files: the license file (license-path), and the key of encryption at rest (encryption.key-file, or your key service). Back them up with your configuration management. A restored data file that was written with encryption at rest cannot be read without its key.

nodes.conf matters as much as the data. It carries the node’s identity, so a node restored without it is a new node (see A node loses nodes.conf). Keep it with the data files, taken at the same time.

Persistence settings first

A backup can only contain what a node wrote down. Decide how much you can lose:

appendonly yes with appendfsync everysec (the default) writes every change to the AOF and flushes it once a second, so a crash of the host loses about a second. appendfsync always flushes every write, and no leaves it to the operating system.
Snapshots (save points, BGSAVE) are what you copy for backups. A node with appendonly no and no save points keeps data only in memory.
WAITAOF numlocal numreplicas timeout makes a client wait until its write is on disk locally and on that many replicas. On a primary with appendonly yes and one replica that also has it on:

redis-cli -p 7500 WAITAOF 1 1 2000

1
1

Use WAITAOF for the writes you must not lose, and WAIT when replica copies are enough. The data on a replica is a copy of its primary’s, so a replica can also be the node you back up.

Take a backup

Ask every node for a snapshot, wait until each has finished, then copy the files. Run the commands from /var/lib/hotkv, where the node directories are:

for port in 7500 7501 7502 7503 7504 7505; do redis-cli -p $port BGSAVE; done

Background saving started
Background saving started
Background saving started
Background saving started
Background saving started
Background saving started

for port in 7500 7501 7502 7503 7504 7505; do
  redis-cli -p $port INFO persistence | grep -E 'rdb_bgsave_in_progress|rdb_last_bgsave_status' | tr '\n' ' '; echo
done

rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok
rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok
rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok
rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok
rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok
rdb_bgsave_in_progress:0 rdb_last_bgsave_status:ok

A snapshot is finished when rdb_bgsave_in_progress is 0 and rdb_last_bgsave_status is ok. Copy dump.hkv, nodes.conf and, when you run the AOF, appendonly.hkv:

mkdir backup
for port in 7500 7501 7502 7503 7504 7505; do
  mkdir backup/$port
  cp $port/dump.hkv $port/appendonly.hkv $port/nodes.conf backup/$port/
done
ls backup/7500

appendonly.hkv
dump.hkv
nodes.conf

Then move backup off the machine. Each node’s files are consistent with that node, but the files of different nodes are taken a moment apart, so writes made during the backup can be in one shard’s files and not another’s. If you need the same moment everywhere, stop your writers while the snapshots run.

Which file a node loads. A node started with appendonly yes that has an AOF loads the AOF, even when a snapshot is older or newer. A node that has only a snapshot loads the snapshot, whether appendonly is yes or no. So to restore from a snapshot alone, remove the node’s old appendonly.hkv first, or restore the AOF from the same backup.

To spare the primaries, take the files of a replica instead: it holds the same data as its primary. Keep the nodes.conf of every node you will restore, including the primaries’.

Restore a replica

A replica’s data is a copy, so the simplest restore is to let it copy again.

It lost its data files but kept nodes.conf. The node comes back with its identity and copies the data from its primary:

kill $(cat 7503/hotkv.pid); sleep 2
rm 7503/dump.hkv 7503/appendonly.hkv

Start the node with start 7503, then wait for the link and check the size:

until redis-cli -p 7503 INFO replication | grep -q master_link_status:up; do sleep 1; done; redis-cli -p 7503 INFO replication | grep -E "role|master_link_status"; redis-cli -p 7503 DBSIZE

role:slave
master_link_status:up
108

108 was the number of keys on its primary. Check: DBSIZE on the replica equals DBSIZE on its primary, and redis-cli --cluster check reports no errors.

It lost everything, nodes.conf too. It starts as a new, empty node with a new ID, unknown to the cluster:

kill $(cat 7503/hotkv.pid); sleep 2
rm -r 7503; mkdir 7503

Start it with start 7503, and it knows only itself, while the cluster still lists the old ID, now failing and without an address:

$ redis-cli -p 7503 CLUSTER NODES
cbd5cf8775d821dd80deec411712e6f91364b66b 127.0.0.1:7503@17503 myself,master - 0 0 0 connected
$ redis-cli -p 7500 CLUSTER NODES | cut -c1-120
a5a0d70bfb22f7a5272d477b8b644a573b5a3c5c 127.0.0.1:7500@17500 myself,master - 0 0 1 connected 0-5460
2fdb058c1e973de99069115a3fd996755c1ece38 127.0.0.1:7501@17501 master - 0 1791068463156 2 connected 5461-10922
dd2b2ca966953a9bf39e379db924d0f8575a6129 127.0.0.1:7502@17502 master - 0 1791068465159 3 connected 10923-16383
f2f83c7b5907e4f1b9265cfbac6c744cd2176234 :0@0 slave,fail,noaddr 2fdb058c1e973de99069115a3fd996755c1ece38 1791068464157 1
...

(Shortly after the old node stops, its entry may still show its address and the flag slave alone.) Introduce the new node to the cluster and make it a replica of the primary the old one followed: the old entry’s fourth field, once the cluster has flagged it slave,fail. The new node copies the data from there:

redis-cli -p 7500 CLUSTER MEET 127.0.0.1 7503
sleep 3
OLDPRIMARY=$(redis-cli -p 7500 CLUSTER NODES | grep 'slave,fail' | cut -d' ' -f4)
redis-cli -p 7503 CLUSTER REPLICATE $OLDPRIMARY
sleep 6
redis-cli -p 7503 INFO replication | grep -E "role|master_link_status"

OK
OK
role:slave
master_link_status:up

The old entry is still listed:

redis-cli -p 7500 CLUSTER NODES | grep noaddr

f2f83c7b5907e4f1b9265cfbac6c744cd2176234 :0@0 slave,fail,noaddr 2fdb058c1e973de99069115a3fd996755c1ece38 1791068464157 1791068459150 4 disconnected

Remove it by telling every node to forget it:

OLD=$(redis-cli -p 7500 CLUSTER NODES | grep noaddr | cut -d' ' -f1)
for port in 7500 7501 7502 7503 7504 7505; do
  redis-cli -p $port CLUSTER FORGET $OLD
done

OK
OK
OK
ERR Unknown node f2f83c7b5907e4f1b9265cfbac6c744cd2176234

OK
OK

The node that never knew the old ID answers ERR Unknown node ..., which you can ignore (which node says so differs, since the new node may or may not have heard of the old one). Forgotten nodes stay forgotten for cluster-blacklist-ttl seconds (60 by default), so run the loop on all nodes close together. The old entry can also disappear on its own once the new node has joined, so always look before you forget. Check:

redis-cli -p 7500 CLUSTER NODES | grep -c fail
redis-cli --cluster check 127.0.0.1:7500 | tail -n 3

0
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

There are no fail entries, the new node is a slave of the right primary, and the check ends with [OK] All 16384 slots covered.

Replace a primary

Let its replica take over first. A stopped primary is replaced automatically by its replica (see Replicas and failover), or run CLUSTER FAILOVER on the replica to do it on purpose. Once the old primary is only a replica, or gone, treat it as a lost replica: restore it as shown above, or re-create it with the steps for a node that loses nodes.conf.

Restore the whole cluster in place

To roll the entire cluster back to a backup, stop every node, put each node’s files back, and start them. The cluster comes back as it was at the backup, with the same node IDs. First, write some data after the backup so that the restore is visible (these keys are not in the backup):

redis-cli --cluster check 127.0.0.1:7500 | head -n 4

127.0.0.1:7500 (bbfb68f7...) -> 113 keys | 5461 slots | 1 slaves.
127.0.0.1:7502 (98295e66...) -> 126 keys | 5461 slots | 1 slaves.
127.0.0.1:7501 (6974d3c3...) -> 111 keys | 5462 slots | 1 slaves.
[OK] 350 keys in 3 masters.

for port in 7500 7501 7502 7503 7504 7505; do kill $(cat $port/hotkv.pid); done
sleep 3
for port in 7500 7501 7502 7503 7504 7505; do
  rm $port/dump.hkv $port/appendonly.hkv $port/nodes.conf
  cp backup/$port/* $port/
done

Start the six nodes with start, one call for each port. Then wait until the cluster is ready and compare:

until redis-cli -p 7500 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done; redis-cli --cluster check 127.0.0.1:7500 | head -n 4

127.0.0.1:7500 (bbfb68f7...) -> 100 keys | 5461 slots | 1 slaves.
127.0.0.1:7501 (6974d3c3...) -> 92 keys | 5462 slots | 1 slaves.
127.0.0.1:7502 (98295e66...) -> 108 keys | 5461 slots | 1 slaves.
[OK] 300 keys in 3 masters.

redis-cli -c -p 7500 GET key:300
redis-cli -c -p 7500 GET key:350
redis-cli -p 7503 INFO replication | grep -E "role|master_link_status"

value:300
(nil)
role:slave
master_link_status:up

The 300 keys of the backup are back, key:350 (written afterwards) is gone, and the replicas are connected (redis-cli prints the (nil) as an empty line when its output is not a terminal). Check: the key count matches the backup, a sample of keys reads back, CLUSTER INFO says cluster_state:ok, and every replica shows master_link_status:up.

Restore the files of a shard’s primary and replicas from the same backup, and the shards of one cluster from the same backup. To bring back one node of a running cluster, restore it from its replica or primary as shown above.

A node loses nodes.conf

If a node’s data is intact and only nodes.conf is gone, it starts as a new node with a new ID and loads its old data. Here, primary 7500 lost the file while its replica took over its slots:

kill $(cat 7500/hotkv.pid); sleep 2
rm 7500/nodes.conf

After start 7500, the node knows only itself, holds keys and owns no slots, while the cluster lists its old ID as failed:

$ redis-cli -p 7501 CLUSTER NODES | cut -c1-110
cbd5cf8775d821dd80deec411712e6f91364b66b 127.0.0.1:7503@17503 slave 2fdb058c1e973de99069115a3fd996755c1ece38 0
2fdb058c1e973de99069115a3fd996755c1ece38 127.0.0.1:7501@17501 myself,master - 0 0 2 connected 5461-10922
a5a0d70bfb22f7a5272d477b8b644a573b5a3c5c 127.0.0.1:7500@17500 master,fail - 1791068481194 1791068476688 1 disc
2dceabab81d7383df895ef008ee0e605fa3a448d 127.0.0.1:7504@17504 slave dd2b2ca966953a9bf39e379db924d0f8575a6129 0
af4718e90d191585d3f0d4f0eec153e26e1c845c 127.0.0.1:7505@17505 master - 0 1791068497251 7 connected 0-5460
dd2b2ca966953a9bf39e379db924d0f8575a6129 127.0.0.1:7502@17502 master - 0 1791068501256 3 connected 10923-16383
$ redis-cli -p 7500 CLUSTER NODES
f3ae190d891c5b743fd6f717532bed475c7f5b50 127.0.0.1:7500@17500 myself,master - 0 0 0 connected
$ redis-cli -p 7500 DBSIZE
100

The data on the new node is stale: its old shard has a new primary, 7505. The node cannot become a replica while it holds that data:

NEW=$(redis-cli -p 7501 CLUSTER NODES | grep ' master ' | grep 0-5460 | cut -d' ' -f1)
redis-cli -p 7501 CLUSTER MEET 127.0.0.1 7500
sleep 3
redis-cli -p 7500 CLUSTER REPLICATE $NEW
redis-cli -p 7500 FLUSHALL
redis-cli -p 7500 CLUSTER REPLICATE $NEW

OK
ERR To set a master the node must be empty and without assigned slots.

OK
OK

After FLUSHALL the replicate works, and the node copies the shard’s current data from the new primary. Then forget the old ID on every node, as above (here the failed entry is the one with noaddr, and the new node, which never knew it, answers ERR Unknown node). Check:

redis-cli -p 7500 INFO replication | grep -E "role|master_link_status"
redis-cli -p 7500 DBSIZE
redis-cli --cluster check 127.0.0.1:7501 | tail -n 4

role:slave
master_link_status:up
100
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

If you lose nodes.conf from every node, restore them from your backup. A cluster whose nodes.conf files are all gone has to be created again with redis-cli --cluster create on empty nodes, and its data loaded from your snapshots or with a migration tool.

Move the cluster to new addresses

To move a cluster to new hosts, or to change the addresses of its nodes, stop every node, copy each node’s directory to its new place, and fix the addresses. nodes.conf records every node’s address, and a node derives its own from its settings but learns the others from the file. The example moves a cluster from 127.0.0.1 to 127.0.0.2 (a different address on the same machine).

Edit nodes.conf. Take a fresh snapshot, stop the nodes, copy, and replace the old address with the new one in each copied nodes.conf:

for port in 7500 7501 7502 7503 7504 7505; do
  redis-cli -p $port BGSAVE
done
sleep 3
for port in 7500 7501 7502 7503 7504 7505; do kill $(cat $port/hotkv.pid); done
sleep 3
mkdir new
for port in 7500 7501 7502 7503 7504 7505; do
  mkdir new/$port
  cp $port/dump.hkv $port/appendonly.hkv $port/nodes.conf new/$port/
  sed -i 's/127\.0\.0\.1/127.0.0.2/g' new/$port/nodes.conf
done
grep -c 127.0.0.2 new/7500/nodes.conf

Background saving started
Background saving started
Background saving started
Background saving started
Background saving started
Background saving started
6

Start the nodes from the new directories, with their new bind (and any new announce settings), then check. With the function above, that is start $port 127.0.0.2 new/$port for each port:

redis-cli -h 127.0.0.2 -p 7500 CLUSTER INFO | head -n 2
redis-cli --cluster check 127.0.0.2:7500 | head -n 4
redis-cli -c -h 127.0.0.2 -p 7500 GET key:123

cluster_enabled:1
cluster_state:ok
127.0.0.2:7501 (2fdb058c...) -> 92 keys | 5462 slots | 1 slaves.
127.0.0.2:7502 (dd2b2ca9...) -> 108 keys | 5461 slots | 1 slaves.
127.0.0.2:7505 (af4718e9...) -> 100 keys | 5461 slots | 1 slaves.
[OK] 300 keys in 3 masters.
value:123

Or introduce the new addresses. Without editing the files, the nodes start with the new bind address but still hold the old addresses of their peers: CLUSTER NODES shows the others as disconnected at their old addresses. This example uses 127.0.0.3 as the new address. Tell one node about the others with CLUSTER MEET at their new addresses:

for port in 7501 7502 7503 7504 7505; do
  redis-cli -h 127.0.0.3 -p 7500 CLUSTER MEET 127.0.0.3 $port
done

OK
OK
OK
OK
OK

The other nodes are now known at their new addresses, and redis-cli --cluster check run against the first node looks healthy. But the node you ran the MEET commands on is still known to every other node at its old address, so a client that is sent to it fails:

$ redis-cli -h 127.0.0.3 -p 7501 CLUSTER NODES | grep ":7500@" | cut -c1-110
6833e519e8bf0a3cab238b594189bb7bae1f52a0 127.0.0.1:7500@17500 master - 1791068911554 1791068911432 1 disconnec
$ redis-cli -c -h 127.0.0.3 -p 7501 GET key:123
Could not connect to Redis at 127.0.0.1:7500: Connection refused

Introduce that node from one of the others as well:

redis-cli -h 127.0.0.3 -p 7501 CLUSTER MEET 127.0.0.3 7500
sleep 8
redis-cli -h 127.0.0.3 -p 7501 CLUSTER NODES | grep ":7500@" | cut -c1-110
redis-cli -c -h 127.0.0.3 -p 7501 GET key:123
redis-cli -c -h 127.0.0.3 -p 7502 GET key:1

OK
6833e519e8bf0a3cab238b594189bb7bae1f52a0 127.0.0.3:7500@17500 master - 0 1791068938505 1 connected 0-5460
value:123
value:1

After that, the node lists itself at the new address too.

Editing nodes.conf is the cleaner way, because the cluster is whole as soon as the nodes start, and nothing is left at an old address. Whichever you use, update your clients and firewalls with the new addresses, and run the checks: the key count, a sample of keys, and CLUSTER INFO.

Ask the HotKV docs