Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Scaling a cluster

Last updated
Not on HotKV CloudOn-Premise: no license needed

A cluster grows by adding primaries and moving hash slots to them, and shrinks by moving the slots away and removing the nodes. Adding a node moves no data by itself: a new primary owns no slots until you give it some. This page covers both steps, every way HotKV can move slots, and what to do when a move goes wrong.

The examples use a cluster on 127.0.0.1 with three primaries and three replicas on ports 8000 to 8005, 100,000 keys in it, and two spare nodes on ports 8006 and 8007. Use your own addresses and ports. Commands that need a node ID take it from CLUSTER MYID.

Set up a practice cluster

Section titled “Set up a practice cluster”

To follow the page on one machine, build the cluster the way the quick start does, on ports 8000 to 8005, and fill it with 100,000 keys. The node directories are created in the current directory, which is where the later commands (--dir node-8006) expect them. The fill script, and the write checker below, need the redis package, which the first commands install in a virtual environment:

Terminal window
mkdir scaling-demo && cd scaling-demo
python3 -m venv venv && venv/bin/pip install -q redis
for port in 8000 8001 8002 8003 8004 8005; do
  mkdir node-$port
  hotkv --port $port --bind 127.0.0.1 --cluster-enabled yes --dir "$PWD/node-$port" \
    --logfile hotkv.log --pidfile "$PWD/node-$port/hotkv.pid" --daemonize yes
done
redis-cli --cluster create 127.0.0.1:8000 127.0.0.1:8001 127.0.0.1:8002 \
  127.0.0.1:8003 127.0.0.1:8004 127.0.0.1:8005 --cluster-replicas 1 --cluster-yes > /dev/null
until redis-cli -p 8000 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done
cat > fill.py <<'EOF'
from redis.cluster import RedisCluster


rc = RedisCluster(host="127.0.0.1", port=8000)
for lo in range(0, 100000, 1000):
    rc.mset_nonatomic({f"key:{i}": i for i in range(lo, lo + 1000)})
print(sum(rc.get_redis_connection(n).dbsize() for n in rc.get_primaries()))
EOF
venv/bin/python fill.py
100000

Before you start

Section titled “Before you start”
  • The cluster must be healthy. redis-cli --cluster check has to end with [OK] All nodes agree about slots configuration. and [OK] All 16384 slots covered., and must list no open slots. Moving slots while a node is down or a slot is open makes both worse. See Monitoring a cluster.
  • Check the target has memory for the slots it receives (see Memory on the target).
  • Moving slots reads a lot of data from the source. Do it when the source has room for the extra load, and move a few hundred slots at a time on a busy cluster.
  • Take a backup first if you cannot afford to lose the data. See Backup and restore.

Check that no write is lost

Section titled “Check that no write is lost”

Run the script that follows, which writes numbered keys, while you move slots. It counts the writes the cluster acknowledged, retries the ones that failed, and at the end reads every acknowledged key back. Run it against your own staging cluster before you move slots in production. It needs the redis package: use the virtual environment from the practice cluster, or make one with python3 -m venv venv && venv/bin/pip install redis, and run the script with venv/bin/python.

"""Write numbered keys to a cluster and check that none that was acknowledged is lost.


Usage: python3 writer.py HOST:PORT SECONDS
"""
import sys
import time


from redis.cluster import RedisCluster
from redis.exceptions import RedisError


host, port = sys.argv[1].rsplit(":", 1)
seconds = float(sys.argv[2])




def connect():
    return RedisCluster(host=host, port=int(port), socket_timeout=2,
                        socket_connect_timeout=2, decode_responses=True)




rc = connect()


acked = 0        # writes the cluster acknowledged
errors = {}      # error text -> how many times a write got it
start = last = last_ok = time.time()
last_acked = 0
longest_stall = 0.0


while time.time() - start < seconds:
    try:
        rc.set(f"w:{acked}", acked)
        acked += 1
        now = time.time()
        longest_stall = max(longest_stall, now - last_ok)
        last_ok = now
    except Exception as e:
        key = type(e).__name__
        errors[key] = errors.get(key, 0) + 1
        if not isinstance(e, RedisError):
            # redis-py lost track of a node it was just redirected to (a node
            # that owned no slots when the script started): connect again
            rc = connect()
        time.sleep(0.05)
    now = time.time()
    if now - last >= 1:
        print(f"t={now - start:4.1f}s acknowledged={acked} "
              f"(+{acked - last_acked}/s) errors={sum(errors.values())}", flush=True)
        last, last_acked = now, acked


missing = 0
for lo in range(0, acked, 500):
    keys = [f"w:{i}" for i in range(lo, min(lo + 500, acked))]
    for i, v in zip(range(lo, lo + len(keys)), rc.mget_nonatomic(keys)):
        if v != str(i):
            missing += 1
print(f"acknowledged={acked} errors={sum(errors.values())} {errors} "
      f"longest_stall={longest_stall:.2f}s missing={missing}")
sys.exit(1 if missing else 0)

The last line is the result: missing=0 means every write the cluster acknowledged was still there. A write that fails is counted in errors and tried again. redis-py can also fail with an AttributeError of its own when a redirect names a node it has not seen (for example a new node that owned no slots when the script started), so the script connects again when it meets an error that is not a Redis error. Run it for as long as the move takes, for example for 40 seconds:

$ python3 writer.py 127.0.0.1:8000 40

It prints a progress line every second, and a last line of the form acknowledged=N errors=N {...} longest_stall=Ns missing=0.

Choose a method

Section titled “Choose a method”

Seven ways of moving slots exist. All of them move the keys of the slots in the selected database or databases and the AI data HotKV keeps by slot (see AI engines in a cluster).

Method What clients see Databases moved A target short of memory
redis-cli --cluster reshard or rebalance, before 8.4 A key that has moved answers -ASK; cluster clients follow it Database 0 only The move stops with OOM and leaves a slot open, which --cluster fix finishes
redis-cli 8.4 or later (atomic, when every node reports redis_version 8.4.0 or later) Nothing: the slots change owner in one step All The move goes on and the target ends above maxmemory
valkey-cli --cluster reshard or rebalance -ASK, as with redis-cli before 8.4 All, one database after the other The move stops and leaves a slot open
valkey-cli ... --cluster-use-atomic-slot-migration Nothing All The migration fails and nothing moves
CLUSTER MIGRATESLOTS on the source Nothing All The migration fails and nothing moves
CLUSTER MIGRATION IMPORT on the target Nothing All The move goes on and the target ends above maxmemory
CLUSTER SETSLOT and MIGRATE by hand -ASK The database you select MIGRATE fails with OOM

Use an atomic method when you can: nothing is half-moved at any moment, a failure leaves the slots where they were, and no client ever gets a redirect for a slot in flight. Use the per-key methods when your tools are older than that. A per-key move works on any cluster, but a failure part way leaves a slot open (see When a move fails).

Add a node

Section titled “Add a node”

Start the new node with cluster-enabled yes, like the others, and an empty data directory. A node that already holds data or knows other nodes cannot join.

$ mkdir node-8006
$ hotkv --port 8006 --cluster-enabled yes --dir node-8006 --daemonize yes --logfile hotkv.log

Join it as a primary with redis-cli --cluster add-node NEW EXISTING, where EXISTING is any node of the cluster:

$ redis-cli --cluster add-node 127.0.0.1:8006 127.0.0.1:8000
>>> Adding node 127.0.0.1:8006 to cluster 127.0.0.1:8000
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to 127.0.0.1:8006 to verify there is no functions in it
>>> Send FUNCTION RESTORE to 127.0.0.1:8006
>>> Send CLUSTER MEET to node 127.0.0.1:8006 to make it join the cluster.
[OK] New node added correctly.
$ redis-cli --cluster check 127.0.0.1:8000
127.0.0.1:8000 (b2c19751...) -> 33314 keys | 5461 slots | 1 slaves.
127.0.0.1:8002 (11145421...) -> 33304 keys | 5461 slots | 1 slaves.
127.0.0.1:8006 (41b47a26...) -> 0 keys | 0 slots | 0 slaves.
127.0.0.1:8001 (a124eb11...) -> 33382 keys | 5462 slots | 1 slaves.
[OK] 100000 keys in 4 masters.

The new primary has 0 slots and 0 keys. It serves nothing until you move slots to it (see Move slots). The tool also copies the function libraries of the cluster to the new node.

To add a replica instead, give the ID of the primary it should follow with --cluster-master-id.

$ mkdir node-8007
$ hotkv --port 8007 --cluster-enabled yes --dir node-8007 --daemonize yes --logfile hotkv.log
$ PRIMARY=$(redis-cli -p 8006 CLUSTER MYID); echo $PRIMARY
41b47a26f9cc4afeb31d827b947c5058f8d9a86c
$ redis-cli --cluster add-node 127.0.0.1:8007 127.0.0.1:8000 --cluster-slave --cluster-master-id $PRIMARY
>>> Adding node 127.0.0.1:8007 to cluster 127.0.0.1:8000
...
>>> Send CLUSTER MEET to node 127.0.0.1:8007 to make it join the cluster.
Waiting for the cluster to join


>>> Configure node as replica of 127.0.0.1:8006.
[OK] New node added correctly.
$ redis-cli -p 8007 ROLE
slave
127.0.0.1
8006
connected
0

The tools do nothing you cannot do by hand: CLUSTER MEET joins a node (give it an IP address, not a name), and CLUSTER REPLICATE makes a joined node a replica. See Replicas and failover.

Move slots with redis-cli

Section titled “Move slots with redis-cli”

redis-cli --cluster reshard and rebalance come with Redis. HotKV works with the 8.0 tools, which move keys one by one, and with 8.4 and later, which use the atomic migration when they can (see Atomic migration with redis-cli). They also work with valkey-cli.

reshard

Section titled “reshard”

reshard moves a number of slots from one node to another. Without options it asks how many slots, which node receives them and which nodes give them. Give the answers as options to run it in a script:

$ ID0=$(redis-cli -p 8000 CLUSTER MYID); ID6=$(redis-cli -p 8006 CLUSTER MYID)
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 1000 --cluster-yes
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving slot 997 from 127.0.0.1:8000 to 127.0.0.1:8006: .......................
Moving slot 998 from 127.0.0.1:8000 to 127.0.0.1:8006: ........
Moving slot 999 from 127.0.0.1:8000 to 127.0.0.1:8006: ........................

--cluster-from takes the ID of the node that gives the slots and --cluster-to the ID of the node that receives them. For each slot the tool marks the slot as importing on the target and migrating on the source, copies its keys with MIGRATE, then gives the slot to the target. While a slot is in that state, a client that asks the source for a key that has already moved gets -ASK, see Resharding by hand. Cluster clients follow ASK themselves, so the writer above reports no missing writes.

rebalance

Section titled “rebalance”

rebalance evens out the number of slots per primary, and --cluster-simulate prints the plan without running it:

$ redis-cli --cluster rebalance 127.0.0.1:8000 --cluster-simulate
>>> Performing Cluster Check (using node 127.0.0.1:8000)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4.00
Moving 1366 slots from 127.0.0.1:8001 to 127.0.0.1:8006
Moving 1365 slots from 127.0.0.1:8002 to 127.0.0.1:8006
Moving 365 slots from 127.0.0.1:8000 to 127.0.0.1:8006

(The tool prints a row of # after each line, one for each slot it would move; they are left out here.) Run it without --cluster-simulate to do the moves. After it, every primary holds 4096 slots:

$ redis-cli --cluster check 127.0.0.1:8000 | grep -E "keys|OK|ERR"
127.0.0.1:8000 (b2c19751...) -> 104871 keys | 4096 slots | 1 slaves.
127.0.0.1:8002 (11145421...) -> 104906 keys | 4096 slots | 1 slaves.
127.0.0.1:8006 (41b47a26...) -> 104998 keys | 4096 slots | 1 slaves.
127.0.0.1:8001 (a124eb11...) -> 104852 keys | 4096 slots | 1 slaves.
[OK] 419627 keys in 4 masters.
25.61 keys per slot on average.
[OK] All nodes agree about slots configuration.
[OK] All 16384 slots covered.

Options that matter:

Option Effect
--cluster-use-empty-masters Include primaries that own no slots. Without it a new, empty primary is left out and the tool reports that nothing needs moving.
--cluster-weight ID=N Give a node a share of N instead of 1. A node with weight 2 gets twice the slots of one with weight 1, and weight 0 empties a node.
--cluster-threshold N Do nothing while every node is within N percent of its share (the default is 2).
--cluster-simulate Print the plan and move nothing. 
$ redis-cli --cluster rebalance 127.0.0.1:8000
...
*** No rebalancing needed! All nodes are within the 2.00% threshold.

Rebalancing balances slots, not keys or load. Slots can differ a lot in size, so if a few slots hold most of the data or traffic, find them first (see Balance by load) and move those.

Atomic migration

Section titled “Atomic migration”

The atomic migration is HotKV’s way of moving slots without ASK redirects. The target gets a copy of the slots and every write made to them while the copy runs, and the slots change owner only when it has all of it. If anything fails before that, the slots are still on the source and the target deletes what it received. There are two ways to start one, one from each end, named after the server that introduced them.

Atomic migration with redis-cli

Section titled “Atomic migration with redis-cli”

redis-cli 8.4 and later moves slots with CLUSTER MIGRATION IMPORT instead of MIGRATE when every node of the cluster reports redis_version 8.4.0 or later in INFO. HotKV reports the version in compat-redis-version, which is 8.4.0 by default, so these tools take the atomic path with no option. The command line is the same as before:

$ redis-cli --version
redis-cli 8.10.2
$ redis-cli -p 8000 INFO server | grep -E "^(redis_version|hotkv_version)"
redis_version:8.4.0
hotkv_version:<version>
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID6 --cluster-to $ID0 --cluster-slots 500 --cluster-yes
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving 500 slots from 127.0.0.1:8006 to 127.0.0.1:8000
Waiting for migration task 9742c8629d7719a17c96013e550e89df938b607f to complete.

The tool asks for one task for all 500 slots instead of moving slot by slot.

To make redis-cli fall back to the per-key move, for example because you want the old behavior, set the version back on every node. The same tool then moves slot by slot:

$ redis-cli -p 8000 CONFIG SET compat-redis-version 7.4.0
$ redis-cli -p 8000 INFO server | grep -E "^redis_version"
redis_version:7.4.0
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 500 --cluster-yes
...
Moving slot 499 from 127.0.0.1:8000 to 127.0.0.1:8006: .......................

compat-redis-version also changes the version INFO, HELLO and scripts report to your applications, so set it back to 8.4.0 when you are done.

Atomic migration with valkey-cli

Section titled “Atomic migration with valkey-cli”

valkey-cli --cluster reshard and rebalance move slots key by key, as the others do, until you add --cluster-use-atomic-slot-migration. With it they send CLUSTER MIGRATESLOTS to the source and wait:

$ valkey-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 300 --cluster-yes --cluster-use-atomic-slot-migration
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving slot range 0-299 from 127.0.0.1:8000 to 127.0.0.1:8006 via atomic slot migration.

Without the flag the same command prints Moving slot N from ... to ... for each slot and moves one database after the other.

CLUSTER MIGRATESLOTS

Section titled “CLUSTER MIGRATESLOTS”

CLUSTER MIGRATESLOTS starts the migration yourself, on the node that owns the slots. It takes ranges and a target node ID, and replies OK at once. Several ranges can share a target, and several groups send slots to different targets in one command:

127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 0 99 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3
OK
127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 100 149 200 249 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3
OK
127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 300 399 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3 SLOTSRANGE 400 499 NODE 80eaf747c8dde56484b36ddd5418b3098c6d3828
OK

Each SLOTSRANGE ... NODE ... group is its own migration and runs at the same time as the others. When all of them have finished, the slots have moved:

127.0.0.1:8000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 149
   3) 1) "127.0.0.1"
      2) (integer) 8006
...

CLUSTER MIGRATION IMPORT

Section titled “CLUSTER MIGRATION IMPORT”

CLUSTER MIGRATION IMPORT is the same move started from the receiving end. Send it to the node that is to get the slots. It finds the owner itself and replies with a task ID:

127.0.0.1:8006> CLUSTER MIGRATION IMPORT 100 199
"0fdd612b88e9a3e41e32718ae4111f9b81e98128"
127.0.0.1:8006> CLUSTER MIGRATION STATUS ID 0fdd612b88e9a3e41e32718ae4111f9b81e98128
1)  1) "id"
    2) "0fdd612b88e9a3e41e32718ae4111f9b81e98128"
    3) "slots"
    4) "100-199"
    ...
   11) "state"
   12) "completed"
   ...

A node runs one IMPORT at a time. Use this form from your own tooling, or let redis-cli 8.4 and later use it for you.

Writes made during the copy

Section titled “Writes made during the copy”

Clients keep writing to the slots while the source copies them. Those writes must reach the target before the slots change owner. By default the source holds every write it takes during the copy and sends them once the copy is out, so the target keeps none of them while the copy arrives.

replica-full-sync-buffer-limit, set on the target, makes it keep some. With a limit, the source sends the first writes of the copy at once, and the target holds them, in order, and runs them when the copy is in:

  • The target keeps writes up to the limit. A write that takes it past the limit is kept whole, the target logs Slots sync buffer limit has been reached, and the source holds the rest as it did before, within its client-output-buffer-limit replica class.
  • mem_cluster_slot_migration_input_buffer in INFO memory shows what the target holds now, and mem_cluster_slot_migration_input_buffer_peak the most it held.
  • The target tells the source its limit only when one is set (the default, 0, keeps none and says nothing). A source that does not know the setting, such as Redis 8, ignores it and holds every write as before, so the migration is not refused.
  • It applies to the Redis 8 style migration (CLUSTER MIGRATION IMPORT, which redis-cli 8.4 and later uses). It does not change how a replica syncs from its primary: that is replication.full-sync-buffer-limit.

Here are two imports of the same 100 slots between two nodes while a client keeps writing to those slots. First with the default, on the target:

127.0.0.1:8006> CONFIG GET replica-full-sync-buffer-limit
1) "replica-full-sync-buffer-limit"
2) "0"
127.0.0.1:8006> CLUSTER MIGRATION IMPORT 0 99
"7146739fe77199a67d2893367add3fd041097c94"
127.0.0.1:8006> INFO memory
...
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0

Then, for the slots to move back, with a limit of 64 MB on the node that gets them:

127.0.0.1:8000> CONFIG SET replica-full-sync-buffer-limit 64mb
OK
127.0.0.1:8000> CONFIG GET replica-full-sync-buffer-limit
1) "replica-full-sync-buffer-limit"
2) "67108864"
127.0.0.1:8000> CLUSTER MIGRATION IMPORT 0 99
"f71cbaef8ade2e39199aa75de3d5b192a892ce3c"
127.0.0.1:8000> INFO memory
...
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:1425344

Both imports complete, and the target ends with the keys of the slots and the keys written meanwhile. With the limit set, the target holds the writes in the buffer during the copy (the peak shows how much) and none once the import is done. Count the limit in the memory you leave on the target (see Memory on the target).

Watch a migration

Section titled “Watch a migration”

Look at both ends of a migration. The two commands show the same migrations in different words:

Where What it shows
CLUSTER GETSLOTMIGRATIONS Every migration this node ran or runs as source or target, newest first, with its state and a message when it failed (fields)
CLUSTER MIGRATION STATUS ALL The same migrations as tasks, with state, last_error, retries and how long writes were held (write_pause_ms) (fields)
CLUSTER INFO cluster_slot_migration_active_tasks, the number of migrations in progress, and the trim counters that follow it
INFO mem_cluster_slot_export and mem_cluster_slot_import (memory the migrations hold), and total_net_cluster_slot_export_bytes and total_net_cluster_slot_import_bytes (bytes sent and received)
CLIENT LIST The connection a migration uses to export: flag E for a CLUSTER MIGRATESLOTS export, flag g for a CLUSTER MIGRATION IMPORT export 
$ redis-cli -p 8000 INFO | grep -E "slot_import|slot_export|slot_migration|paused_reason"
paused_reason:none
mem_cluster_slot_import:0
mem_cluster_slot_export:0
mem_cluster_slot_migration_output_buffer:0
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0
total_net_cluster_slot_import_bytes:0
total_net_cluster_slot_export_bytes:344755

(That was taken on a source while it exported.) The per-key methods have no such counters: while a slot is moving, CLUSTER NODES shows it on the source as [slot->-target-id] and on the target as [slot-<-source-id], and redis-cli --cluster check lists the open slots.

Cancel a migration

Section titled “Cancel a migration”

To stop an atomic migration, send CLUSTER CANCELSLOTMIGRATIONS to the source. The slots stay on the source and the target deletes what it received:

127.0.0.1:8000> CLUSTER CANCELSLOTMIGRATIONS
OK
127.0.0.1:8000> CLUSTER COUNTKEYSINSLOT 150
(integer) 8
127.0.0.1:8006> CLUSTER COUNTKEYSINSLOT 150
(integer) 0

A task started with CLUSTER MIGRATION IMPORT is different: if its export is cancelled the target tries again every second. End it from the target with CLUSTER MIGRATION CANCEL:

127.0.0.1:8006> CLUSTER MIGRATION CANCEL ID 9ec1175e94f56fd3e5082beb8011f2d1e3a2c0ab
(integer) 1
127.0.0.1:8006> CLUSTER MIGRATION STATUS ID 9ec1175e94f56fd3e5082beb8011f2d1e3a2c0ab
...
   11) "state"
   12) "canceled"
   13) "last_error"
   14) "Cancelled due to user request"
...

A per-key reshard that is stopped part way, for example by killing redis-cli, leaves the slot it was moving open until you run redis-cli --cluster fix (see When a move fails).

A small data set moves too quickly to cancel by hand. To try this, use a larger data set on a staging cluster.

When a move fails

Section titled “When a move fails”

The atomic methods report why in message (CLUSTER GETSLOTMIGRATIONS) or last_error (CLUSTER MIGRATION STATUS). A migration that fails leaves the slots on the source, and the target deletes what it had received. These are the messages you can meet:

Message Cause and fix
Received error during handshake to target: -NOAUTH Authentication required. The nodes have requirepass and the source has no masterauth. Set masterauth to the same password on every node (CONFIG SET masterauth ...) and start again.
Received error during handshake to target: -ERR Slot migration can only be used on primary nodes. The target is a replica. Name a primary.
Ran out of memory (OOM) during slot import (on the target) and Connection lost to target... (on the source) The target has no room for the slots. See Memory on the target.
Connection lost to source. or Connection lost to target. The other node closed the connection: it was cancelled or restarted, or the network dropped. Check that node, then start again.

valkey-cli prints the source’s side of it as Atomic slot migration failed: .... A task from CLUSTER MIGRATION IMPORT keeps trying: look at retries and last_error, fix the cause and the next try goes through, or cancel it.

The per-key methods fail differently. They stop at the first error, and the slot being moved stays open: importing on the target, migrating on the source. Cluster clients keep working, but redis-cli --cluster check reports it:

$ redis-cli --cluster check 127.0.0.1:8000 | grep -E "WARN|ERR|OK|open"
[OK] 100000 keys in 4 masters.
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 127.0.0.1:8000 has slots in migrating state 10.
[WARNING] Node 127.0.0.1:8007 has slots in importing state 10.
[WARNING] Node 127.0.0.1:8006 has slots in importing state 10.
[WARNING] Node 127.0.0.1:8003 has slots in migrating state 10.
[WARNING] The following slots are open: 10.
[OK] All 16384 slots covered.

Fix the cause, then let the tool finish the slot:

$ redis-cli --cluster fix 127.0.0.1:8000 | tail -n 4
>>> Case 1: Moving slot 10 from 127.0.0.1:8000 to 127.0.0.1:8006
Moving slot 10 from 127.0.0.1:8000 to 127.0.0.1:8006: ...
>>> Check slots coverage...
[OK] All 16384 slots covered.

The slot is also listed on the replicas of both nodes, because they record the state. If redis-cli reports LOADING right after the last slot left a node, the node became a replica of the new owner (see Remove a node) and is copying its data: wait for it, and run the step again.

Memory on the target

Section titled “Memory on the target”

A move adds the size of the slots to the target, and it adds it before the source deletes anything. Check INFO memory on the target (used_memory against maxmemory) and leave room for the slots plus the writes that arrive meanwhile. How a node that is short of room reacts depends on the method. In the examples below the target has a maxmemory only slightly above what it uses when empty, maxmemory-policy noeviction, and receives thousands of keys:

  • redis-cli before 8.4 (and valkey-cli without the flag) stop at the first MIGRATE the target refuses:

    Node 127.0.0.1:8000 replied with error:
    ERR OOM command not allowed when used memory > 'maxmemory'.
    

    clusterManagerMoveSlot failed: ERR OOM command not allowed when used memory > 'maxmemory'.

    The slot is left open, as above.

  • valkey-cli --cluster-use-atomic-slot-migration and CLUSTER MIGRATESLOTS fail and move nothing. The tool says:

    Atomic slot migration failed: Connection lost to target. Check CLUSTER GETSLOTMIGRATIONS on the target node for more information.

    and the target shows the reason:

    127.0.0.1:8006> CLUSTER GETSLOTMIGRATIONS
    ...
    17) "state"
    18) "failed"
    19) "message"
    20) "Ran out of memory (OOM) during slot import"
    ...

  • redis-cli 8.4 and later and CLUSTER MIGRATION IMPORT do not stop: the task completes and the target ends far above its limit. With noeviction the target then refuses writes until it is below the limit again:

    127.0.0.1:8006> SET k2 x
    (error) OOM command not allowed when used memory > 'maxmemory'.
    $ redis-cli -p 8006 INFO memory | grep -E "^(used_memory|maxmemory):"
    used_memory:9379664
    maxmemory:2299080

    Free memory by raising maxmemory or by moving slots on to another node.

In practice: size maxmemory on every primary for the data it will hold after the move, not before, and check used_memory on the target first. A target with replica-full-sync-buffer-limit set needs room for that many bytes of writes as well.

Several databases

Section titled “Several databases”

A cluster node has one database unless you set cluster-databases (1 to 128, at startup only). With 1, SELECT of another database, MOVE and COPY ... DB are refused. With more, how a move treats the other databases depends on the method. Take cluster-databases 2 with keys in each database, and move 500 slots:

Method Result
redis-cli before 8.4 Moved database 0 only and stopped at the first slot that had keys in database 1: clusterManagerMoveSlot failed: ERR Can't assign hashslot 2 to a different node while I still hold keys for this hash slot. The slot stayed open.
valkey-cli, with and without --cluster-use-atomic-slot-migration Moved both databases
redis-cli 8.4 or later (atomic) Moved both databases
CLUSTER MIGRATESLOTS, CLUSTER MIGRATION IMPORT Move every database 
$ for p in 8000 8001 8002 8006; do printf "%s  db0=%s db1=%s\n" $p $(redis-cli -p $p -n 0 DBSIZE) $(redis-cli -p $p -n 1 DBSIZE); done
8000  db0=908 db1=908
8001  db0=988 db1=993
8002  db0=1009 db1=1010
8006  db0=95 db1=89

(That is the layout after a valkey-cli move: the target holds keys in both databases.) So a cluster with more than one database needs valkey-cli, or redis-cli 8.4 or later with the atomic path.

Balance by load

Section titled “Balance by load”

CLUSTER SLOT-STATS shows what each slot of a node holds and costs, so you can move the slots that matter instead of an even number of slots. It reports key count always, and CPU time, network bytes and an estimate of memory when cluster-slot-stats-enabled says so. Memory can only be turned on at startup, so start the nodes with cluster-slot-stats-enabled yes (all three) if you want it.

In this example 3000 keys of 1000 bytes sit under one hash tag ({report}, slot 4432) and one key ({hot}:counter, slot 6093) is read 100,000 times. Find the largest slot by key count or memory on one node, and the busiest on another:

127.0.0.1:8000> CLUSTER SLOT-STATS ORDERBY key-count LIMIT 2
1) 1) (integer) 4432
   2)  1) "key-count"
       2) (integer) 3000
       3) "memory-bytes"
       4) (integer) 3294000
       5) "cpu-usec"
       6) (integer) 2052
       7) "network-bytes-in"
       8) (integer) 3124890
       9) "network-bytes-out"
      10) (integer) 3139890
2) 1) (integer) 28
   2)  1) "key-count"
       2) (integer) 4
...
127.0.0.1:8001> CLUSTER SLOT-STATS ORDERBY cpu-usec LIMIT 2
1) 1) (integer) 6093
   2)  1) "key-count"
       2) (integer) 1
       3) "memory-bytes"
       4) (integer) 97
       5) "cpu-usec"
       6) (integer) 840
       7) "network-bytes-in"
       8) (integer) 3300040
       9) "network-bytes-out"
      10) (integer) 700045
2) 1) (integer) 8242
...

Each node reports only the slots of its own shard, so ask every primary. Then move the slot you found. Here the hot slot goes to the empty node, and its counters start again from zero on the new owner:

127.0.0.1:8001> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
...
127.0.0.1:8001> CLUSTER MIGRATESLOTS SLOTSRANGE 6093 6093 NODE 1130ae68cfca9762086e6e2fb09277bf32adc689
OK
127.0.0.1:8006> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
1) 1) (integer) 6093
   2)  1) "key-count"
       2) (integer) 1
       3) "memory-bytes"
       4) (integer) 97
       5) "cpu-usec"
       6) (integer) 0
       7) "network-bytes-in"
       8) (integer) 0
       9) "network-bytes-out"
      10) (integer) 0
127.0.0.1:8001> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
(empty array)

Keys that share a hash tag live in one slot and always move together, and a single hot key cannot be split across nodes: moving its slot moves the load, it does not spread it. See Keys and commands.

Resharding by hand

Section titled “Resharding by hand”

This is what redis-cli does for one slot, and what to run to repair a slot it left open. The key {cart1}:a is in slot 3311, on the primary of port 8000, and three keys share its hash tag. (On a cluster that holds other keys, a slot holds them too: CLUSTER GETKEYSINSLOT lists every key of the slot, and MIGRATE ... KEYS must be given all of them before SETSLOT ... NODE is accepted.)

127.0.0.1:8006> CLUSTER SETSLOT 3311 IMPORTING c1591dee05ff2f4d974468fa936f59c4a9a26411
OK
127.0.0.1:8000> CLUSTER SETSLOT 3311 MIGRATING 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8000> CLUSTER GETKEYSINSLOT 3311 100
1) "{cart1}:a"
2) "{cart1}:b"
3) "{cart1}:c"
127.0.0.1:8000> MIGRATE 127.0.0.1 8006 {cart1}:a 0 5000
OK

IMPORTING goes to the target and names the source, and MIGRATING goes to the source and names the target. Now one key has moved and two have not, and the source tells a client which is which:

127.0.0.1:8000> GET {cart1}:a
(error) ASK 3311 127.0.0.1:8006
127.0.0.1:8000> GET {cart1}:b
"2"
127.0.0.1:8000> MGET {cart1}:a {cart1}:b
(error) TRYAGAIN Multiple keys request during rehashing of slot
127.0.0.1:8006> GET {cart1}:a
(error) MOVED 3311 127.0.0.1:8000
$ printf "ASKING\nGET {cart1}:a\n" | redis-cli --no-raw -p 8006
OK
"1"

ASK tells the client to ask the target once, after ASKING, and not to update its slot map. The target answers MOVED to a client that did not send ASKING, because it does not own the slot yet. A command with several keys, some moved and some not, gets TRYAGAIN. Move the rest of the keys, then give the slot to the target on both nodes:

127.0.0.1:8000> MIGRATE 127.0.0.1 8006 "" 0 5000 KEYS {cart1}:b {cart1}:c
OK
127.0.0.1:8000> CLUSTER COUNTKEYSINSLOT 3311
(integer) 0
127.0.0.1:8000> CLUSTER SETSLOT 3311 NODE 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8006> CLUSTER SETSLOT 3311 NODE 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8000> GET {cart1}:a
(error) MOVED 3311 127.0.0.1:8006

CLUSTER SETSLOT refuses to run on a slot while an atomic migration of it runs (ERR Slot export in progress. on the source, ERR Slot import in progress. on the target), and CLUSTER MIGRATESLOTS refuses while a slot is open by hand. Close the open slot with CLUSTER SETSLOT slot STABLE first.

Remove a node

Section titled “Remove a node”

A node can leave only when it owns no slots. Move them away, then remove the node.

redis-cli --cluster del-node refuses while the node still owns slots:

$ redis-cli --cluster del-node 127.0.0.1:8000 $ID6
>>> Removing node 6ee8e6e776c5c96fead4ed430a2ae206c67e5285 from cluster 127.0.0.1:8000
[ERR] Node 127.0.0.1:8006 is not empty! Reshard data away and try again.

Empty it with rebalance and a weight of 0, or with reshard. Here is the first:

$ redis-cli --cluster rebalance 127.0.0.1:8000 --cluster-weight $ID6=0 --cluster-use-empty-masters
...
>>> Rebalancing across 4 nodes. Total weight = 3.00
Moving 1000 slots from 127.0.0.1:8006 to 127.0.0.1:8000
Moving 1 slots from 127.0.0.1:8001 to 127.0.0.1:8000

When a primary gives away its last slot, it becomes a replica of the node that received it (the setting cluster-allow-replica-migration, on by default), and its own replica follows it there:

$ redis-cli -p 8000 CLUSTER NODES | grep -E "8006|8007" | cut -d" " -f2,3,4,8
127.0.0.1:8007@18007 slave 050f4c13e4aa7473a73a4defed5fad0d4b413a14 connected
127.0.0.1:8006@18006 slave 050f4c13e4aa7473a73a4defed5fad0d4b413a14 connected

Remove replicas first, then the empty node. del-node sends CLUSTER FORGET to every other node and CLUSTER RESET SOFT to the removed one:

$ redis-cli --cluster del-node 127.0.0.1:8000 $ID7
>>> Removing node f904377c2be8fb5f8058b18a104e17e89ffa12d2 from cluster 127.0.0.1:8000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
$ redis-cli --cluster del-node 127.0.0.1:8000 $ID6
>>> Removing node 6ee8e6e776c5c96fead4ed430a2ae206c67e5285 from cluster 127.0.0.1:8000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
$ redis-cli -p 8000 CLUSTER NODES | wc -l
6
$ redis-cli -p 8006 CLUSTER NODES
6ee8e6e776c5c96fead4ed430a2ae206c67e5285 127.0.0.1:8006@18006 myself,master - 0 0 7 connected

The removed node still runs, alone and empty. Stop it with SHUTDOWN NOSAVE and delete its directory, or reuse it.

To do it by hand, send CLUSTER FORGET with the node’s ID to every other node within 60 seconds (cluster-blacklist-ttl): during those 60 seconds a node that forgot it ignores what the others say about it, so gossip cannot bring it back, and after that it can. A replica cannot forget its own primary. Then reset the removed node with CLUSTER RESET HARD so it forgets the cluster too (a primary must hold no keys for that):

127.0.0.1:8000> CLUSTER MEET 127.0.0.1 8006
OK
127.0.0.1:8000> CLUSTER FORGET f4b848b7bdff9b9354a046cd1483a80f6bc40298
OK
127.0.0.1:8001> CLUSTER FORGET f4b848b7bdff9b9354a046cd1483a80f6bc40298
OK
...
127.0.0.1:8006> CLUSTER RESET HARD
OK
$ redis-cli -p 8006 CLUSTER NODES | cut -d" " -f2,3,8,9
127.0.0.1:8006@18006 myself,master connected

Clear a slot or a range

Section titled “Clear a slot or a range”

CLUSTER FLUSHSLOT deletes the keys of one slot, and SFLUSH the slots of the ranges you give, without touching the rest of the node. Use them to drop a slot’s data, not to move it.

Next steps

Section titled “Next steps”