Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Upgrading a cluster

Last updated
Not on HotKV CloudOn-Premise: no license needed

A cluster with a replica for every primary can be upgraded while it serves traffic. You upgrade the replicas, hand each shard over to its upgraded replica, and then upgrade the old primaries, which are replicas by then. No setting has to change while you work. This page walks through that procedure on a running cluster. For installing a package, the data files and the license, see Upgrading.

The cluster in the examples has six nodes on 127.0.0.1, ports 8200 to 8205, three primaries and three replicas, with --appendonly yes. In the commands, hotkv is the new binary.

Before you start

Section titled “Before you start”

  • Every primary needs a replica. Without one, the shard is unavailable while its primary restarts. Add replicas first (see Replicas and failover).

  • Each node needs persistence (an AOF or snapshots) and must keep its data directory. A node comes back from its directory with the same ID and its data. Its nodes.conf keeps its identity:

    $ redis-cli -p 8211 CLUSTER MYID
    2b3677d1bd3474cc2abfe7188e5961d092fed174
    $ redis-cli -p 8211 SHUTDOWN
    $ hotkv --port 8211 --cluster-enabled yes --dir node-8211 --daemonize yes --logfile hotkv.log --appendonly yes
    $ redis-cli -p 8211 CLUSTER MYID
    2b3677d1bd3474cc2abfe7188e5961d092fed174
    $ redis-cli -p 8211 INFO server | grep hotkv_version
    hotkv_version:<version>

  • Take a backup of every node (see Backup and restore) and keep the old packages. A rollback needs the data from before the upgrade.

  • Read the changelog for the new version.

  • The cluster must be healthy: redis-cli --cluster check ends in [OK] All nodes agree about slots configuration. and [OK] All 16384 slots covered..

The procedure

Section titled “The procedure”

Upgrade a node by stopping it, installing the new version, and starting it with the same configuration and data directory. Stop it with SHUTDOWN or SIGTERM so it saves its data. Do the steps in this order. The first listing is the state before:

$ redis-cli -p 8200 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort
127.0.0.1:8200@18200 myself,master - connected 0-5460
127.0.0.1:8201@18201 master - connected 5461-10922
127.0.0.1:8202@18202 master - connected 10923-16383
127.0.0.1:8203@18203 slave 0f0f77b88856a00b042e9c88b99c0e7c03a5ea0b connected
127.0.0.1:8204@18204 slave 18a5b6634156c6094c07e06bc219e9a527f8597e connected
127.0.0.1:8205@18205 slave 907081b33dd5d1611de69b29e2c0cec612dcfa1b connected

Keep the time you spend with two versions in one cluster short, and do not use settings that the new version introduces until every node has it. Do not move slots while the versions are mixed: finish the upgrade first.

1. The replicas, one at a time

Section titled “1. The replicas, one at a time”

Stop one replica, start it on the new version, and wait until it is a replica again with its link up. Check the cluster before the next one:

$ redis-cli -p 8204 SHUTDOWN
$ hotkv --port 8204 --cluster-enabled yes --dir node-8204 --daemonize yes --logfile hotkv.log --appendonly yes
$ redis-cli -p 8204 INFO replication | grep -E '^(role|master_link_status)'
role:slave
master_link_status:up
$ redis-cli -p 8204 INFO server | grep ^hotkv_version
hotkv_version:<version>
$ redis-cli -p 8200 CLUSTER INFO | grep -E 'cluster_state|cluster_slots_ok'
cluster_state:ok
cluster_slots_ok:16384

Repeat for every replica. If master_link_status stays down, look at the log of the replica and of its primary before you go on: a link that does not come up means the next step would have nothing to promote.

2. Each primary, through its replica

Section titled “2. Each primary, through its replica”

Run CLUSTER FAILOVER on the upgraded replica of each shard, one shard at a time. Use the default mode, not FORCE or TAKEOVER. It pauses the primary’s writes for a moment, lets the replica catch up, and swaps the roles. Wait until the replica is a primary before you start the next shard:

$ for r in 8204 8205 8203; do redis-cli -p $r CLUSTER FAILOVER; until redis-cli -p $r ROLE | head -n 1 | grep -q master; do sleep 1; done; done
OK
OK
OK
$ redis-cli -p 8200 CLUSTER NODES | cut -d" " -f2,3,8,9 | sort
127.0.0.1:8200@18200 myself,slave connected
127.0.0.1:8201@18201 slave connected
127.0.0.1:8202@18202 slave connected
127.0.0.1:8203@18203 master connected 10923-16383
127.0.0.1:8204@18204 master connected 0-5460
127.0.0.1:8205@18205 master connected 5461-10922

The old primaries, still on the old version, are replicas of the new ones. Writes to a shard wait for the handover and then go to the new primary: they are parked, not failed.

Do not send the CLUSTER FAILOVER commands together. If one fails to win its election, its log says Manual failover timed out. It did not win an election in time, the shard keeps its old primary, and stopping what you take for the old primaries then restarts the wrong nodes. After each handover, check the roles with CLUSTER NODES or ROLE, as the loop does.

Do not stop an old primary before its shard has been handed over: check the roles first. A primary that comes back without its data (no AOF and no snapshot) is empty, and its replicas copy the emptiness.

3. The old primaries

Section titled “3. The old primaries”

They are replicas now. Upgrade them as in step 1:

$ redis-cli -p 8200 SHUTDOWN
$ hotkv --port 8200 --cluster-enabled yes --dir node-8200 --daemonize yes --logfile hotkv.log --appendonly yes
$ redis-cli -p 8200 INFO replication | grep -E '^(role|master_link_status)'
role:slave
master_link_status:up

When the last node is done, every node reports the new version and the cluster is whole:

$ for p in $(seq 8200 8205); do printf "%s %s\n" $p "$(redis-cli -p $p INFO server | tr -d '\r' | grep ^hotkv_version)"; done
8200 hotkv_version:<version>
8201 hotkv_version:<version>
8202 hotkv_version:<version>
8203 hotkv_version:<version>
8204 hotkv_version:<version>
8205 hotkv_version:<version>
$ redis-cli --cluster check 127.0.0.1:8200 | grep -E "OK|ERR|WARN"
[OK] 266569 keys in 3 masters.
[OK] All nodes agree about slots configuration.
[OK] All 16384 slots covered.

The shards now have their primaries on the former replicas. That is fine. If you want the old layout back, run CLUSTER FAILOVER on each former primary.

What clients see

Section titled “What clients see”

A cluster client that follows MOVED sees no errors during a handover, only a short pause while the shard changes primary. A client that does not follow MOVED, or that keeps a connection to one node, sees errors for as long as that node is down. You can check this on your own staging cluster by running writer.py from Scaling a cluster for the whole upgrade: missing=0 at the end means no acknowledged write was lost.

Next steps

Section titled “Next steps”