# Upgrading a cluster

> Upgrade a HotKV cluster one node at a time without taking it down, and what to check between steps.

URL: https://docs.hotkv.com/clustering/upgrades/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

A cluster with a replica for every primary can be upgraded while it serves traffic.
You upgrade the replicas, hand each shard over to its upgraded replica, and then
upgrade the old primaries, which are replicas by then. No setting has to change while
you work. This page walks through that procedure on a running cluster. For installing
a package, the data files and the license, see [Upgrading](https://docs.hotkv.com/operate/upgrading/index.md).

The cluster in the examples has six nodes on `127.0.0.1`, ports 8200 to 8205, three
primaries and three replicas, with `--appendonly yes`. In the commands, `hotkv` is the
new binary.

## Before you start

- Every primary needs a replica. Without one, the shard is unavailable while its
  primary restarts. Add replicas first (see [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md#add-a-replica)).
- Each node needs persistence (an AOF or snapshots) and must keep its data directory.
  A node comes back from its directory with the same ID and its data. Its
  `nodes.conf` keeps its identity:

  ```
  $ redis-cli -p 8211 CLUSTER MYID
  2b3677d1bd3474cc2abfe7188e5961d092fed174
  $ redis-cli -p 8211 SHUTDOWN
  $ hotkv --port 8211 --cluster-enabled yes --dir node-8211 --daemonize yes --logfile hotkv.log --appendonly yes
  $ redis-cli -p 8211 CLUSTER MYID
  2b3677d1bd3474cc2abfe7188e5961d092fed174
  $ redis-cli -p 8211 INFO server | grep hotkv_version
  hotkv_version:<version>
  ```
- Take a backup of every node (see [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md))
  and keep the old packages. A rollback needs the data from before the upgrade.
- Read the [changelog](https://docs.hotkv.com/changelog/index.md) for the new version.
- The cluster must be healthy: `redis-cli --cluster check` ends in `[OK] All nodes agree
  about slots configuration.` and `[OK] All 16384 slots covered.`.

## The procedure

Upgrade a node by stopping it, installing the new version, and starting it with the
same configuration and data directory. Stop it with `SHUTDOWN` or `SIGTERM` so it
saves its data. Do the steps in this order. The first listing is the state before:

```
$ redis-cli -p 8200 CLUSTER NODES | cut -d" " -f2,3,4,8,9 | sort
127.0.0.1:8200@18200 myself,master - connected 0-5460
127.0.0.1:8201@18201 master - connected 5461-10922
127.0.0.1:8202@18202 master - connected 10923-16383
127.0.0.1:8203@18203 slave 0f0f77b88856a00b042e9c88b99c0e7c03a5ea0b connected
127.0.0.1:8204@18204 slave 18a5b6634156c6094c07e06bc219e9a527f8597e connected
127.0.0.1:8205@18205 slave 907081b33dd5d1611de69b29e2c0cec612dcfa1b connected
```

Keep the time you spend with two versions in one cluster short, and do not use
settings that the new version introduces until every node has it. Do not move slots
while the versions are mixed: finish the upgrade first.

### 1. The replicas, one at a time

Stop one replica, start it on the new version, and wait until it is a replica again
with its link up. Check the cluster before the next one:

```
$ redis-cli -p 8204 SHUTDOWN
$ hotkv --port 8204 --cluster-enabled yes --dir node-8204 --daemonize yes --logfile hotkv.log --appendonly yes
$ redis-cli -p 8204 INFO replication | grep -E '^(role|master_link_status)'
role:slave
master_link_status:up
$ redis-cli -p 8204 INFO server | grep ^hotkv_version
hotkv_version:<version>
$ redis-cli -p 8200 CLUSTER INFO | grep -E 'cluster_state|cluster_slots_ok'
cluster_state:ok
cluster_slots_ok:16384
```

Repeat for every replica. If `master_link_status` stays `down`, look at the log of
the replica and of its primary before you go on: a link that does not come up
means the next step would have nothing to promote.

### 2. Each primary, through its replica

Run `CLUSTER FAILOVER` on the upgraded replica of each shard, one shard at a time.
Use the default mode, not `FORCE` or `TAKEOVER`. It pauses the primary's writes for a
moment, lets the replica catch up, and swaps the roles. Wait until the replica is a
primary before you start the next shard:

```
$ for r in 8204 8205 8203; do redis-cli -p $r CLUSTER FAILOVER; until redis-cli -p $r ROLE | head -n 1 | grep -q master; do sleep 1; done; done
OK
OK
OK
$ redis-cli -p 8200 CLUSTER NODES | cut -d" " -f2,3,8,9 | sort
127.0.0.1:8200@18200 myself,slave connected
127.0.0.1:8201@18201 slave connected
127.0.0.1:8202@18202 slave connected
127.0.0.1:8203@18203 master connected 10923-16383
127.0.0.1:8204@18204 master connected 0-5460
127.0.0.1:8205@18205 master connected 5461-10922
```

The old primaries, still on the old version, are replicas of the new ones. Writes to a
shard wait for the handover and then go to the new primary: they are parked, not
failed.

Do not send the `CLUSTER FAILOVER` commands together. If one fails to win its election,
its log says `Manual failover timed out. It did not win an election in time`, the shard
keeps its old primary, and stopping what you take for the old primaries then restarts
the wrong nodes. After each handover, check the roles with `CLUSTER NODES` or `ROLE`,
as the loop does.

Do not stop an old primary before its shard has been handed over: check the roles first.
A primary that comes back without its data (no AOF and no snapshot) is empty, and its
replicas copy the emptiness.

### 3. The old primaries

They are replicas now. Upgrade them as in step 1:

```
$ redis-cli -p 8200 SHUTDOWN
$ hotkv --port 8200 --cluster-enabled yes --dir node-8200 --daemonize yes --logfile hotkv.log --appendonly yes
$ redis-cli -p 8200 INFO replication | grep -E '^(role|master_link_status)'
role:slave
master_link_status:up
```

When the last node is done, every node reports the new version and the cluster is whole:

```
$ for p in $(seq 8200 8205); do printf "%s %s\n" $p "$(redis-cli -p $p INFO server | tr -d '\r' | grep ^hotkv_version)"; done
8200 hotkv_version:<version>
8201 hotkv_version:<version>
8202 hotkv_version:<version>
8203 hotkv_version:<version>
8204 hotkv_version:<version>
8205 hotkv_version:<version>
$ redis-cli --cluster check 127.0.0.1:8200 | grep -E "OK|ERR|WARN"
[OK] 266569 keys in 3 masters.
[OK] All nodes agree about slots configuration.
[OK] All 16384 slots covered.
```

The shards now have their primaries on the former replicas. That is fine. If you want the
old layout back, run `CLUSTER FAILOVER` on each former primary.

### What clients see

A cluster client that follows `MOVED` sees no errors during a handover, only a short
pause while the shard changes primary. A client that does not follow `MOVED`, or that
keeps a connection to one node, sees errors for as long as that node is down. You can
check this on your own staging cluster by running `writer.py` from
[Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md#check-that-no-write-is-lost) for the whole
upgrade: `missing=0` at the end means no acknowledged write was lost.

## Next steps

- [Upgrading](https://docs.hotkv.com/operate/upgrading/index.md): packages, containers, the license and the data
  files.
- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md#start-a-failover-yourself):
  the commands this procedure uses.
- [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md): the checks to run between steps.