# Scaling a cluster

> Add and remove nodes, move slots with redis-cli, valkey-cli or the atomic migration commands, and watch, cancel and recover a migration.

URL: https://docs.hotkv.com/clustering/scaling/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

A cluster grows by adding primaries and moving hash slots to them, and shrinks
by moving the slots away and removing the nodes. Adding a node moves no data by
itself: a new primary owns no slots until you give it some. This page covers
both steps, every way HotKV can move slots, and what to do when a move goes
wrong.

The examples use a cluster on `127.0.0.1` with three primaries and three
replicas on ports 8000 to 8005, 100,000 keys in it, and two spare nodes on
ports 8006 and 8007. Use your own addresses and ports. Commands that need a
node ID take it from `CLUSTER MYID`.

## Set up a practice cluster

To follow the page on one machine, build the cluster the way the
[quick start](https://docs.hotkv.com/clustering/quick-start/index.md) does, on ports 8000 to 8005, and fill it
with 100,000 keys. The node directories are created in the current directory,
which is where the later commands (`--dir node-8006`) expect them. The fill
script, and the write checker below, need the `redis` package, which the first
commands install in a virtual environment:

```bash
mkdir scaling-demo && cd scaling-demo
python3 -m venv venv && venv/bin/pip install -q redis
for port in 8000 8001 8002 8003 8004 8005; do
  mkdir node-$port
  hotkv --port $port --bind 127.0.0.1 --cluster-enabled yes --dir "$PWD/node-$port" \
    --logfile hotkv.log --pidfile "$PWD/node-$port/hotkv.pid" --daemonize yes
done
redis-cli --cluster create 127.0.0.1:8000 127.0.0.1:8001 127.0.0.1:8002 \
  127.0.0.1:8003 127.0.0.1:8004 127.0.0.1:8005 --cluster-replicas 1 --cluster-yes > /dev/null
until redis-cli -p 8000 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done
cat > fill.py <<'EOF'
from redis.cluster import RedisCluster

rc = RedisCluster(host="127.0.0.1", port=8000)
for lo in range(0, 100000, 1000):
    rc.mset_nonatomic({f"key:{i}": i for i in range(lo, lo + 1000)})
print(sum(rc.get_redis_connection(n).dbsize() for n in rc.get_primaries()))
EOF
venv/bin/python fill.py
```

```text
100000
```

## Before you start

- The cluster must be healthy. `redis-cli --cluster check` has to end with
  `[OK] All nodes agree about slots configuration.` and `[OK] All 16384 slots
  covered.`, and must list no open slots. Moving slots while a node is down or
  a slot is open makes both worse. See [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md).
- Check the target has memory for the slots it receives (see
  [Memory on the target](#memory-on-the-target)).
- Moving slots reads a lot of data from the source. Do it when the source has
  room for the extra load, and move a few hundred slots at a time on a busy
  cluster.
- Take a backup first if you cannot afford to lose the data. See
  [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md).

### Check that no write is lost

Run the script that follows, which writes numbered keys, while you move slots.
It counts the writes the cluster acknowledged, retries the ones that failed, and
at the end reads every acknowledged key back. Run it against your own staging
cluster before you move slots in production. It needs the `redis` package: use the
virtual environment from the practice cluster, or make one with
`python3 -m venv venv && venv/bin/pip install redis`, and run the script with
`venv/bin/python`.

```python
"""Write numbered keys to a cluster and check that none that was acknowledged is lost.

Usage: python3 writer.py HOST:PORT SECONDS
"""
import sys
import time

from redis.cluster import RedisCluster
from redis.exceptions import RedisError

host, port = sys.argv[1].rsplit(":", 1)
seconds = float(sys.argv[2])

def connect():
    return RedisCluster(host=host, port=int(port), socket_timeout=2,
                        socket_connect_timeout=2, decode_responses=True)

rc = connect()

acked = 0        # writes the cluster acknowledged
errors = {}      # error text -> how many times a write got it
start = last = last_ok = time.time()
last_acked = 0
longest_stall = 0.0

while time.time() - start < seconds:
    try:
        rc.set(f"w:{acked}", acked)
        acked += 1
        now = time.time()
        longest_stall = max(longest_stall, now - last_ok)
        last_ok = now
    except Exception as e:
        key = type(e).__name__
        errors[key] = errors.get(key, 0) + 1
        if not isinstance(e, RedisError):
            # redis-py lost track of a node it was just redirected to (a node
            # that owned no slots when the script started): connect again
            rc = connect()
        time.sleep(0.05)
    now = time.time()
    if now - last >= 1:
        print(f"t={now - start:4.1f}s acknowledged={acked} "
              f"(+{acked - last_acked}/s) errors={sum(errors.values())}", flush=True)
        last, last_acked = now, acked

missing = 0
for lo in range(0, acked, 500):
    keys = [f"w:{i}" for i in range(lo, min(lo + 500, acked))]
    for i, v in zip(range(lo, lo + len(keys)), rc.mget_nonatomic(keys)):
        if v != str(i):
            missing += 1
print(f"acknowledged={acked} errors={sum(errors.values())} {errors} "
      f"longest_stall={longest_stall:.2f}s missing={missing}")
sys.exit(1 if missing else 0)
```

The last line is the result: `missing=0` means every write the cluster
acknowledged was still there. A write that fails is counted in `errors` and tried
again. redis-py can also fail with an `AttributeError` of its own when a redirect
names a node it has not seen (for example a new node that owned no slots when
the script started), so the script connects again when it meets an error that
is not a Redis error. Run it for as long as the move takes, for example for 40
seconds:

```
$ python3 writer.py 127.0.0.1:8000 40
```

It prints a progress line every second, and a last line of the form
`acknowledged=N errors=N {...} longest_stall=Ns missing=0`.

## Choose a method

Seven ways of moving slots exist. All of them move the keys of the slots in the
selected database or databases and the AI data HotKV keeps by slot (see
[AI engines in a cluster](https://docs.hotkv.com/clustering/ai-engines/index.md)).

| Method | What clients see | Databases moved | A target short of memory |
| --- | --- | --- | --- |
| `redis-cli --cluster reshard` or `rebalance`, before 8.4 | A key that has moved answers `-ASK`; cluster clients follow it | Database 0 only | The move stops with `OOM` and leaves a slot open, which `--cluster fix` finishes |
| `redis-cli` 8.4 or later (atomic, when every node reports `redis_version` 8.4.0 or later) | Nothing: the slots change owner in one step | All | The move goes on and the target ends above `maxmemory` |
| `valkey-cli --cluster reshard` or `rebalance` | `-ASK`, as with `redis-cli` before 8.4 | All, one database after the other | The move stops and leaves a slot open |
| `valkey-cli ... --cluster-use-atomic-slot-migration` | Nothing | All | The migration fails and nothing moves |
| [`CLUSTER MIGRATESLOTS`](https://docs.hotkv.com/commands/cluster-migrateslots/index.md) on the source | Nothing | All | The migration fails and nothing moves |
| [`CLUSTER MIGRATION IMPORT`](https://docs.hotkv.com/commands/cluster-migration/index.md) on the target | Nothing | All | The move goes on and the target ends above `maxmemory` |
| `CLUSTER SETSLOT` and `MIGRATE` by hand | `-ASK` | The database you select | `MIGRATE` fails with `OOM` |

Use an atomic method when you can: nothing is half-moved at any moment, a
failure leaves the slots where they were, and no client ever gets a redirect
for a slot in flight. Use the per-key methods when your tools are older than
that. A per-key move works on any cluster, but a failure part
way leaves a slot open (see [When a move fails](#when-a-move-fails)).

> **Caution**
> A cluster whose nodes keep data in more than one database cannot be moved with
> `redis-cli` before 8.4: it moves database 0 and stops at the first slot that
> holds keys in another one. See [Several databases](#several-databases).

## Add a node

Start the new node with `cluster-enabled yes`, like the others, and an empty
data directory. A node that already holds data or knows other nodes cannot
join.

```
$ mkdir node-8006
$ hotkv --port 8006 --cluster-enabled yes --dir node-8006 --daemonize yes --logfile hotkv.log
```

Join it as a primary with `redis-cli --cluster add-node NEW EXISTING`, where
`EXISTING` is any node of the cluster:

```
$ redis-cli --cluster add-node 127.0.0.1:8006 127.0.0.1:8000
>>> Adding node 127.0.0.1:8006 to cluster 127.0.0.1:8000
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to 127.0.0.1:8006 to verify there is no functions in it
>>> Send FUNCTION RESTORE to 127.0.0.1:8006
>>> Send CLUSTER MEET to node 127.0.0.1:8006 to make it join the cluster.
[OK] New node added correctly.
$ redis-cli --cluster check 127.0.0.1:8000
127.0.0.1:8000 (b2c19751...) -> 33314 keys | 5461 slots | 1 slaves.
127.0.0.1:8002 (11145421...) -> 33304 keys | 5461 slots | 1 slaves.
127.0.0.1:8006 (41b47a26...) -> 0 keys | 0 slots | 0 slaves.
127.0.0.1:8001 (a124eb11...) -> 33382 keys | 5462 slots | 1 slaves.
[OK] 100000 keys in 4 masters.
```

The new primary has 0 slots and 0 keys. It serves nothing until you move slots to
it (see [Move slots](#move-slots-with-redis-cli)). The tool also copies the
function libraries of the cluster to the new node.

To add a replica instead, give the ID of the primary it should follow with
`--cluster-master-id`.

```
$ mkdir node-8007
$ hotkv --port 8007 --cluster-enabled yes --dir node-8007 --daemonize yes --logfile hotkv.log
$ PRIMARY=$(redis-cli -p 8006 CLUSTER MYID); echo $PRIMARY
41b47a26f9cc4afeb31d827b947c5058f8d9a86c
$ redis-cli --cluster add-node 127.0.0.1:8007 127.0.0.1:8000 --cluster-slave --cluster-master-id $PRIMARY
>>> Adding node 127.0.0.1:8007 to cluster 127.0.0.1:8000
...
>>> Send CLUSTER MEET to node 127.0.0.1:8007 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 127.0.0.1:8006.
[OK] New node added correctly.
$ redis-cli -p 8007 ROLE
slave
127.0.0.1
8006
connected
0
```

The tools do nothing you cannot do by hand: [`CLUSTER MEET`](https://docs.hotkv.com/commands/cluster-meet/index.md)
joins a node (give it an IP address, not a name), and
[`CLUSTER REPLICATE`](https://docs.hotkv.com/commands/cluster-replicate/index.md) makes a joined node a
replica. See [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md#add-a-replica).

## Move slots with redis-cli

`redis-cli --cluster reshard` and `rebalance` come with Redis. HotKV works with
the 8.0 tools, which move keys one by one, and with 8.4 and later, which use the
atomic migration when they can (see [Atomic migration with
redis-cli](#atomic-migration-with-redis-cli)). They also work with `valkey-cli`.

### reshard

`reshard` moves a number of slots from one node to another. Without options it
asks how many slots, which node receives them and which nodes give them.
Give the answers as options to run it in a script:

```
$ ID0=$(redis-cli -p 8000 CLUSTER MYID); ID6=$(redis-cli -p 8006 CLUSTER MYID)
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 1000 --cluster-yes
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving slot 997 from 127.0.0.1:8000 to 127.0.0.1:8006: .......................
Moving slot 998 from 127.0.0.1:8000 to 127.0.0.1:8006: ........
Moving slot 999 from 127.0.0.1:8000 to 127.0.0.1:8006: ........................
```

`--cluster-from` takes the ID of the node that gives the slots and `--cluster-to`
the ID of the node that receives them. For each slot the tool marks the slot as
importing on the target and migrating on the source, copies its keys with
[`MIGRATE`](https://docs.hotkv.com/commands/migrate/index.md), then gives the slot to the target. While a slot
is in that state, a client that asks the source for a key that has already moved
gets `-ASK`, see [Resharding by hand](#resharding-by-hand). Cluster clients
follow `ASK` themselves, so the writer above reports no missing writes.

### rebalance

`rebalance` evens out the number of slots per primary, and `--cluster-simulate`
prints the plan without running it:

```
$ redis-cli --cluster rebalance 127.0.0.1:8000 --cluster-simulate
>>> Performing Cluster Check (using node 127.0.0.1:8000)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4.00
Moving 1366 slots from 127.0.0.1:8001 to 127.0.0.1:8006
Moving 1365 slots from 127.0.0.1:8002 to 127.0.0.1:8006
Moving 365 slots from 127.0.0.1:8000 to 127.0.0.1:8006
```

(The tool prints a row of `#` after each line, one for each slot it would move; they
are left out here.) Run it without `--cluster-simulate` to do the moves. After
it, every primary holds 4096 slots:

```
$ redis-cli --cluster check 127.0.0.1:8000 | grep -E "keys|OK|ERR"
127.0.0.1:8000 (b2c19751...) -> 104871 keys | 4096 slots | 1 slaves.
127.0.0.1:8002 (11145421...) -> 104906 keys | 4096 slots | 1 slaves.
127.0.0.1:8006 (41b47a26...) -> 104998 keys | 4096 slots | 1 slaves.
127.0.0.1:8001 (a124eb11...) -> 104852 keys | 4096 slots | 1 slaves.
[OK] 419627 keys in 4 masters.
25.61 keys per slot on average.
[OK] All nodes agree about slots configuration.
[OK] All 16384 slots covered.
```

Options that matter:

| Option | Effect |
| --- | --- |
| `--cluster-use-empty-masters` | Include primaries that own no slots. Without it a new, empty primary is left out and the tool reports that nothing needs moving. |
| `--cluster-weight ID=N` | Give a node a share of N instead of 1. A node with weight 2 gets twice the slots of one with weight 1, and weight 0 empties a node. |
| `--cluster-threshold N` | Do nothing while every node is within N percent of its share (the default is 2). |
| `--cluster-simulate` | Print the plan and move nothing. |

```
$ redis-cli --cluster rebalance 127.0.0.1:8000
...
*** No rebalancing needed! All nodes are within the 2.00% threshold.
```

Rebalancing balances slots, not keys or load. Slots can differ a lot in size, so
if a few slots hold most of the data or traffic, find them first (see
[Balance by load](#balance-by-load)) and move those.

## Atomic migration

The atomic migration is HotKV's way of moving slots without `ASK` redirects. The
target gets a copy of the slots and every write made to them while the copy runs,
and the slots change owner only when it has all of it. If anything fails before
that, the slots are still on the source and the target deletes what it received.
There are two ways to start one, one from each end, named after the server that
introduced them.

### Atomic migration with redis-cli

`redis-cli` 8.4 and later moves slots with `CLUSTER MIGRATION IMPORT` instead of
`MIGRATE` when every node of the cluster reports `redis_version` 8.4.0 or later in
`INFO`. HotKV reports the version in `compat-redis-version`, which is `8.4.0` by
default, so these tools take the atomic path with no option. The command line
is the same as before:

```
$ redis-cli --version
redis-cli 8.10.2
$ redis-cli -p 8000 INFO server | grep -E "^(redis_version|hotkv_version)"
redis_version:8.4.0
hotkv_version:<version>
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID6 --cluster-to $ID0 --cluster-slots 500 --cluster-yes
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving 500 slots from 127.0.0.1:8006 to 127.0.0.1:8000
Waiting for migration task 9742c8629d7719a17c96013e550e89df938b607f to complete.
```

The tool asks for one task for all 500 slots instead of moving slot by slot.

To make `redis-cli` fall back to the per-key move, for example because you want
the old behavior, set the version back on every node. The same tool then moves
slot by slot:

```
$ redis-cli -p 8000 CONFIG SET compat-redis-version 7.4.0
$ redis-cli -p 8000 INFO server | grep -E "^redis_version"
redis_version:7.4.0
$ redis-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 500 --cluster-yes
...
Moving slot 499 from 127.0.0.1:8000 to 127.0.0.1:8006: .......................
```

`compat-redis-version` also changes the version `INFO`, `HELLO` and scripts
report to your applications, so set it back to `8.4.0` when you are done.

### Atomic migration with valkey-cli

`valkey-cli --cluster reshard` and `rebalance` move slots key by key, as the
others do, until you add `--cluster-use-atomic-slot-migration`. With it they send
`CLUSTER MIGRATESLOTS` to the source and wait:

```
$ valkey-cli --cluster reshard 127.0.0.1:8000 --cluster-from $ID0 --cluster-to $ID6 --cluster-slots 300 --cluster-yes --cluster-use-atomic-slot-migration
>>> Performing Cluster Check (using node 127.0.0.1:8000)
...
Moving slot range 0-299 from 127.0.0.1:8000 to 127.0.0.1:8006 via atomic slot migration.
```

Without the flag the same command prints `Moving slot N from ... to ...` for each
slot and moves one database after the other.

### CLUSTER MIGRATESLOTS

[`CLUSTER MIGRATESLOTS`](https://docs.hotkv.com/commands/cluster-migrateslots/index.md) starts the migration
yourself, on the node that owns the slots. It takes ranges and a target node ID,
and replies `OK` at once. Several ranges can share a target, and several groups
send slots to different targets in one command:

```
127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 0 99 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3
OK
127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 100 149 200 249 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3
OK
127.0.0.1:8000> CLUSTER MIGRATESLOTS SLOTSRANGE 300 399 NODE 49a04a859cbe2f18b168fdff41f4f2346045f9e3 SLOTSRANGE 400 499 NODE 80eaf747c8dde56484b36ddd5418b3098c6d3828
OK
```

Each `SLOTSRANGE ... NODE ...` group is its own migration and runs at the same time
as the others. When all of them have finished, the slots have moved:

```
127.0.0.1:8000> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 149
   3) 1) "127.0.0.1"
      2) (integer) 8006
...
```

### CLUSTER MIGRATION IMPORT

[`CLUSTER MIGRATION IMPORT`](https://docs.hotkv.com/commands/cluster-migration/index.md) is the same move started
from the receiving end. Send it to the node that is to get the slots. It finds the
owner itself and replies with a task ID:

```
127.0.0.1:8006> CLUSTER MIGRATION IMPORT 100 199
"0fdd612b88e9a3e41e32718ae4111f9b81e98128"
127.0.0.1:8006> CLUSTER MIGRATION STATUS ID 0fdd612b88e9a3e41e32718ae4111f9b81e98128
1)  1) "id"
    2) "0fdd612b88e9a3e41e32718ae4111f9b81e98128"
    3) "slots"
    4) "100-199"
    ...
   11) "state"
   12) "completed"
   ...
```

A node runs one `IMPORT` at a time. Use this form from your own tooling, or let
`redis-cli` 8.4 and later use it for you.

### Writes made during the copy

Clients keep writing to the slots while the source copies them. Those writes must
reach the target before the slots change owner. By default the source holds every
write it takes during the copy and sends them once the copy is out, so the target
keeps none of them while the copy arrives.

[`replica-full-sync-buffer-limit`](https://docs.hotkv.com/reference/configuration/index.md#replica-full-sync-buffer-limit),
set on the **target**, makes it keep some. With a limit, the source sends the first
writes of the copy at once, and the target holds them, in order, and runs them when
the copy is in:

- The target keeps writes up to the limit. A write that takes it past the limit is
  kept whole, the target logs `Slots sync buffer limit has been reached`, and the
  source holds the rest as it did before, within its
  [`client-output-buffer-limit`](https://docs.hotkv.com/reference/configuration/index.md#client-output-buffer-limit)
  `replica` class.
- `mem_cluster_slot_migration_input_buffer` in `INFO memory` shows what the target
  holds now, and `mem_cluster_slot_migration_input_buffer_peak` the most it held.
- The target tells the source its limit only when one is set (the default, `0`,
  keeps none and says nothing). A source that does not know the setting, such as
  Redis 8, ignores it and holds every write as before, so the migration is not
  refused.
- It applies to the Redis 8 style migration (`CLUSTER MIGRATION IMPORT`, which
  `redis-cli` 8.4 and later uses). It does not change how a replica syncs from its
  primary: that is [`replication.full-sync-buffer-limit`](https://docs.hotkv.com/reference/configuration/index.md#replication.full-sync-buffer-limit).

Here are two imports of the same 100 slots between two nodes while a client keeps
writing to those slots. First with the default, on the target:

```
127.0.0.1:8006> CONFIG GET replica-full-sync-buffer-limit
1) "replica-full-sync-buffer-limit"
2) "0"
127.0.0.1:8006> CLUSTER MIGRATION IMPORT 0 99
"7146739fe77199a67d2893367add3fd041097c94"
127.0.0.1:8006> INFO memory
...
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0
```

Then, for the slots to move back, with a limit of 64 MB on the node that gets them:

```
127.0.0.1:8000> CONFIG SET replica-full-sync-buffer-limit 64mb
OK
127.0.0.1:8000> CONFIG GET replica-full-sync-buffer-limit
1) "replica-full-sync-buffer-limit"
2) "67108864"
127.0.0.1:8000> CLUSTER MIGRATION IMPORT 0 99
"f71cbaef8ade2e39199aa75de3d5b192a892ce3c"
127.0.0.1:8000> INFO memory
...
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:1425344
```

Both imports complete, and the target ends with the keys of the slots and the keys
written meanwhile. With the limit set, the target holds the writes in the buffer
during the copy (the peak shows how much) and none once the import is done.
Count the limit in the memory you leave on the target (see
[Memory on the target](#memory-on-the-target)).

## Watch a migration

Look at both ends of a migration. The two commands show the same migrations in
different words:

| Where | What it shows |
| --- | --- |
| `CLUSTER GETSLOTMIGRATIONS` | Every migration this node ran or runs as source or target, newest first, with its `state` and a `message` when it failed ([fields](https://docs.hotkv.com/commands/cluster-getslotmigrations/index.md)) |
| `CLUSTER MIGRATION STATUS ALL` | The same migrations as tasks, with `state`, `last_error`, `retries` and how long writes were held (`write_pause_ms`) ([fields](https://docs.hotkv.com/commands/cluster-migration/index.md)) |
| `CLUSTER INFO` | `cluster_slot_migration_active_tasks`, the number of migrations in progress, and the trim counters that follow it |
| `INFO` | `mem_cluster_slot_export` and `mem_cluster_slot_import` (memory the migrations hold), and `total_net_cluster_slot_export_bytes` and `total_net_cluster_slot_import_bytes` (bytes sent and received) |
| `CLIENT LIST` | The connection a migration uses to export: flag `E` for a `CLUSTER MIGRATESLOTS` export, flag `g` for a `CLUSTER MIGRATION IMPORT` export |

```
$ redis-cli -p 8000 INFO | grep -E "slot_import|slot_export|slot_migration|paused_reason"
paused_reason:none
mem_cluster_slot_import:0
mem_cluster_slot_export:0
mem_cluster_slot_migration_output_buffer:0
mem_cluster_slot_migration_input_buffer:0
mem_cluster_slot_migration_input_buffer_peak:0
total_net_cluster_slot_import_bytes:0
total_net_cluster_slot_export_bytes:344755
```

(That was taken on a source while it exported.) The per-key methods have no such
counters: while a slot is moving, `CLUSTER NODES` shows it on the source as
`[slot->-target-id]` and on the target as `[slot-<-source-id]`, and
`redis-cli --cluster check` lists the open slots.

## Cancel a migration

To stop an atomic migration, send
[`CLUSTER CANCELSLOTMIGRATIONS`](https://docs.hotkv.com/commands/cluster-cancelslotmigrations/index.md) to the
source. The slots stay on the source and the target deletes what it received:

```
127.0.0.1:8000> CLUSTER CANCELSLOTMIGRATIONS
OK
127.0.0.1:8000> CLUSTER COUNTKEYSINSLOT 150
(integer) 8
127.0.0.1:8006> CLUSTER COUNTKEYSINSLOT 150
(integer) 0
```

A task started with `CLUSTER MIGRATION IMPORT` is different: if its export is
cancelled the target tries again every second. End it from the target with
[`CLUSTER MIGRATION CANCEL`](https://docs.hotkv.com/commands/cluster-migration/index.md):

```
127.0.0.1:8006> CLUSTER MIGRATION CANCEL ID 9ec1175e94f56fd3e5082beb8011f2d1e3a2c0ab
(integer) 1
127.0.0.1:8006> CLUSTER MIGRATION STATUS ID 9ec1175e94f56fd3e5082beb8011f2d1e3a2c0ab
...
   11) "state"
   12) "canceled"
   13) "last_error"
   14) "Cancelled due to user request"
...
```

A per-key reshard that is stopped part way, for example by killing `redis-cli`,
leaves the slot it was moving open until you run `redis-cli --cluster fix` (see
[When a move fails](#when-a-move-fails)).

A small data set moves too quickly to cancel by hand. To try this, use a larger
data set on a staging cluster.

## When a move fails

The atomic methods report why in `message` (`CLUSTER GETSLOTMIGRATIONS`) or
`last_error` (`CLUSTER MIGRATION STATUS`). A migration that fails leaves the
slots on the source, and the target deletes what it had received. These are the
messages you can meet:

| Message | Cause and fix |
| --- | --- |
| `Received error during handshake to target: -NOAUTH Authentication required.` | The nodes have `requirepass` and the source has no `masterauth`. Set `masterauth` to the same password on every node (`CONFIG SET masterauth ...`) and start again. |
| `Received error during handshake to target: -ERR Slot migration can only be used on primary nodes.` | The target is a replica. Name a primary. |
| `Ran out of memory (OOM) during slot import` (on the target) and `Connection lost to target...` (on the source) | The target has no room for the slots. See [Memory on the target](#memory-on-the-target). |
| `Connection lost to source.` or `Connection lost to target.` | The other node closed the connection: it was cancelled or restarted, or the network dropped. Check that node, then start again. |

`valkey-cli` prints the source's side of it as `Atomic slot migration failed: ...`.
A task from `CLUSTER MIGRATION IMPORT` keeps trying: look at `retries` and
`last_error`, fix the cause and the next try goes through, or cancel it.

The per-key methods fail differently. They stop at the first error, and the slot
being moved stays open: importing on the target, migrating on the source. Cluster
clients keep working, but `redis-cli --cluster check` reports it:

```
$ redis-cli --cluster check 127.0.0.1:8000 | grep -E "WARN|ERR|OK|open"
[OK] 100000 keys in 4 masters.
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 127.0.0.1:8000 has slots in migrating state 10.
[WARNING] Node 127.0.0.1:8007 has slots in importing state 10.
[WARNING] Node 127.0.0.1:8006 has slots in importing state 10.
[WARNING] Node 127.0.0.1:8003 has slots in migrating state 10.
[WARNING] The following slots are open: 10.
[OK] All 16384 slots covered.
```

Fix the cause, then let the tool finish the slot:

```
$ redis-cli --cluster fix 127.0.0.1:8000 | tail -n 4
>>> Case 1: Moving slot 10 from 127.0.0.1:8000 to 127.0.0.1:8006
Moving slot 10 from 127.0.0.1:8000 to 127.0.0.1:8006: ...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

The slot is also listed on the replicas of both nodes, because they record the
state. If `redis-cli` reports `LOADING` right after the last slot left a node,
the node became a replica of the new owner (see [Remove a node](#remove-a-node))
and is copying its data: wait for it, and run the step again.

## Memory on the target

A move adds the size of the slots to the target, and it adds it before the source
deletes anything. Check `INFO memory` on the target (`used_memory` against
`maxmemory`) and leave room for the slots plus the writes that arrive meanwhile.
How a node that is short of room reacts depends on the method. In the examples
below the target has a `maxmemory` only slightly above what it uses when empty,
`maxmemory-policy noeviction`, and receives thousands of keys:

- `redis-cli` before 8.4 (and `valkey-cli` without the flag) stop at the first
  `MIGRATE` the target refuses:

  ```
  Node 127.0.0.1:8000 replied with error:
  ERR OOM command not allowed when used memory > 'maxmemory'.

  clusterManagerMoveSlot failed: ERR OOM command not allowed when used memory > 'maxmemory'.
  ```

  The slot is left open, as above.

- `valkey-cli --cluster-use-atomic-slot-migration` and `CLUSTER MIGRATESLOTS` fail
  and move nothing. The tool says:

  ```
  Atomic slot migration failed: Connection lost to target. Check CLUSTER GETSLOTMIGRATIONS on the target node for more information.
  ```

  and the target shows the reason:

  ```
  127.0.0.1:8006> CLUSTER GETSLOTMIGRATIONS
  ...
  17) "state"
  18) "failed"
  19) "message"
  20) "Ran out of memory (OOM) during slot import"
  ...
  ```

- `redis-cli` 8.4 and later and `CLUSTER MIGRATION IMPORT` do not stop: the task
  completes and the target ends far above its limit. With `noeviction` the target
  then refuses writes until it is below the limit again:

  ```
  127.0.0.1:8006> SET k2 x
  (error) OOM command not allowed when used memory > 'maxmemory'.
  $ redis-cli -p 8006 INFO memory | grep -E "^(used_memory|maxmemory):"
  used_memory:9379664
  maxmemory:2299080
  ```

  Free memory by raising `maxmemory` or by moving slots on to another node.

In practice: size `maxmemory` on every primary for the data it will hold after the
move, not before, and check `used_memory` on the target first. A target with
[`replica-full-sync-buffer-limit`](#writes-made-during-the-copy) set needs room for
that many bytes of writes as well.

## Several databases

A cluster node has one database unless you set `cluster-databases` (1 to 128, at
startup only). With 1, `SELECT` of another database, `MOVE` and `COPY ... DB` are
refused. With more, how a move treats the other databases depends on the method.
Take `cluster-databases 2` with keys in each database, and move 500 slots:

| Method | Result |
| --- | --- |
| `redis-cli` before 8.4 | Moved database 0 only and stopped at the first slot that had keys in database 1: `clusterManagerMoveSlot failed: ERR Can't assign hashslot 2 to a different node while I still hold keys for this hash slot.` The slot stayed open. |
| `valkey-cli`, with and without `--cluster-use-atomic-slot-migration` | Moved both databases |
| `redis-cli` 8.4 or later (atomic) | Moved both databases |
| `CLUSTER MIGRATESLOTS`, `CLUSTER MIGRATION IMPORT` | Move every database |

```
$ for p in 8000 8001 8002 8006; do printf "%s  db0=%s db1=%s\n" $p $(redis-cli -p $p -n 0 DBSIZE) $(redis-cli -p $p -n 1 DBSIZE); done
8000  db0=908 db1=908
8001  db0=988 db1=993
8002  db0=1009 db1=1010
8006  db0=95 db1=89
```

(That is the layout after a `valkey-cli` move: the target holds keys in both databases.) So a cluster with more than one
database needs `valkey-cli`, or `redis-cli` 8.4 or later with the atomic path.

## Balance by load

`CLUSTER SLOT-STATS` shows what each slot of a node holds and costs, so you can
move the slots that matter instead of an even number of slots. It reports key count
always, and CPU time, network bytes and an estimate of memory when
`cluster-slot-stats-enabled` says so. Memory can only be turned on at startup, so
start the nodes with `cluster-slot-stats-enabled yes` (all three) if you want it.

In this example 3000 keys of 1000 bytes sit under one hash tag (`{report}`, slot 4432)
and one key (`{hot}:counter`, slot 6093) is read 100,000 times. Find the largest slot
by key count or memory on one node, and the busiest on another:

```
127.0.0.1:8000> CLUSTER SLOT-STATS ORDERBY key-count LIMIT 2
1) 1) (integer) 4432
   2)  1) "key-count"
       2) (integer) 3000
       3) "memory-bytes"
       4) (integer) 3294000
       5) "cpu-usec"
       6) (integer) 2052
       7) "network-bytes-in"
       8) (integer) 3124890
       9) "network-bytes-out"
      10) (integer) 3139890
2) 1) (integer) 28
   2)  1) "key-count"
       2) (integer) 4
...
127.0.0.1:8001> CLUSTER SLOT-STATS ORDERBY cpu-usec LIMIT 2
1) 1) (integer) 6093
   2)  1) "key-count"
       2) (integer) 1
       3) "memory-bytes"
       4) (integer) 97
       5) "cpu-usec"
       6) (integer) 840
       7) "network-bytes-in"
       8) (integer) 3300040
       9) "network-bytes-out"
      10) (integer) 700045
2) 1) (integer) 8242
...
```

Each node reports only the slots of its own shard, so ask every primary. Then move
the slot you found. Here the hot slot goes to the empty node, and its counters
start again from zero on the new owner:

```
127.0.0.1:8001> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
...
127.0.0.1:8001> CLUSTER MIGRATESLOTS SLOTSRANGE 6093 6093 NODE 1130ae68cfca9762086e6e2fb09277bf32adc689
OK
127.0.0.1:8006> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
1) 1) (integer) 6093
   2)  1) "key-count"
       2) (integer) 1
       3) "memory-bytes"
       4) (integer) 97
       5) "cpu-usec"
       6) (integer) 0
       7) "network-bytes-in"
       8) (integer) 0
       9) "network-bytes-out"
      10) (integer) 0
127.0.0.1:8001> CLUSTER SLOT-STATS SLOTSRANGE 6093 6093
(empty array)
```

Keys that share a hash tag live in one slot and always move together, and a
single hot key cannot be split across nodes: moving its slot moves the load, it
does not spread it. See [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md).

## Resharding by hand

This is what `redis-cli` does for one slot, and what to run to repair a slot it
left open. The key `{cart1}:a` is in slot 3311, on the primary of port 8000, and
three keys share its hash tag. (On a cluster that holds other keys, a slot holds
them too: `CLUSTER GETKEYSINSLOT` lists every key of the slot, and `MIGRATE ... KEYS`
must be given all of them before `SETSLOT ... NODE` is accepted.)

```
127.0.0.1:8006> CLUSTER SETSLOT 3311 IMPORTING c1591dee05ff2f4d974468fa936f59c4a9a26411
OK
127.0.0.1:8000> CLUSTER SETSLOT 3311 MIGRATING 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8000> CLUSTER GETKEYSINSLOT 3311 100
1) "{cart1}:a"
2) "{cart1}:b"
3) "{cart1}:c"
127.0.0.1:8000> MIGRATE 127.0.0.1 8006 {cart1}:a 0 5000
OK
```

`IMPORTING` goes to the target and names the source, and `MIGRATING` goes to the
source and names the target. Now one key has moved and two have not, and the
source tells a client which is which:

```
127.0.0.1:8000> GET {cart1}:a
(error) ASK 3311 127.0.0.1:8006
127.0.0.1:8000> GET {cart1}:b
"2"
127.0.0.1:8000> MGET {cart1}:a {cart1}:b
(error) TRYAGAIN Multiple keys request during rehashing of slot
127.0.0.1:8006> GET {cart1}:a
(error) MOVED 3311 127.0.0.1:8000
$ printf "ASKING\nGET {cart1}:a\n" | redis-cli --no-raw -p 8006
OK
"1"
```

`ASK` tells the client to ask the target once, after `ASKING`, and not to update
its slot map. The target answers `MOVED` to a client that did not send `ASKING`,
because it does not own the slot yet. A command with several keys, some moved and
some not, gets `TRYAGAIN`. Move the rest of the keys, then give the slot to the
target on both nodes:

```
127.0.0.1:8000> MIGRATE 127.0.0.1 8006 "" 0 5000 KEYS {cart1}:b {cart1}:c
OK
127.0.0.1:8000> CLUSTER COUNTKEYSINSLOT 3311
(integer) 0
127.0.0.1:8000> CLUSTER SETSLOT 3311 NODE 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8006> CLUSTER SETSLOT 3311 NODE 7a21031eb3404d8598386ed737fa18fdf5db0750
OK
127.0.0.1:8000> GET {cart1}:a
(error) MOVED 3311 127.0.0.1:8006
```

[`CLUSTER SETSLOT`](https://docs.hotkv.com/commands/cluster-setslot/index.md) refuses to run on a slot while an
atomic migration of it runs (`ERR Slot export in progress.` on the source, `ERR Slot
import in progress.` on the target), and `CLUSTER MIGRATESLOTS` refuses while a slot
is open by hand. Close the open slot with `CLUSTER SETSLOT slot STABLE` first.

## Remove a node

A node can leave only when it owns no slots. Move them away, then remove the node.

`redis-cli --cluster del-node` refuses while the node still owns slots:

```
$ redis-cli --cluster del-node 127.0.0.1:8000 $ID6
>>> Removing node 6ee8e6e776c5c96fead4ed430a2ae206c67e5285 from cluster 127.0.0.1:8000
[ERR] Node 127.0.0.1:8006 is not empty! Reshard data away and try again.
```

Empty it with `rebalance` and a weight of 0, or with `reshard`. Here is the first:

```
$ redis-cli --cluster rebalance 127.0.0.1:8000 --cluster-weight $ID6=0 --cluster-use-empty-masters
...
>>> Rebalancing across 4 nodes. Total weight = 3.00
Moving 1000 slots from 127.0.0.1:8006 to 127.0.0.1:8000
Moving 1 slots from 127.0.0.1:8001 to 127.0.0.1:8000
```

When a primary gives away its last slot, it becomes a replica of the node that
received it (the setting `cluster-allow-replica-migration`, on by default), and
its own replica follows it there:

```
$ redis-cli -p 8000 CLUSTER NODES | grep -E "8006|8007" | cut -d" " -f2,3,4,8
127.0.0.1:8007@18007 slave 050f4c13e4aa7473a73a4defed5fad0d4b413a14 connected
127.0.0.1:8006@18006 slave 050f4c13e4aa7473a73a4defed5fad0d4b413a14 connected
```

Remove replicas first, then the empty node. `del-node` sends `CLUSTER FORGET` to
every other node and `CLUSTER RESET SOFT` to the removed one:

```
$ redis-cli --cluster del-node 127.0.0.1:8000 $ID7
>>> Removing node f904377c2be8fb5f8058b18a104e17e89ffa12d2 from cluster 127.0.0.1:8000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
$ redis-cli --cluster del-node 127.0.0.1:8000 $ID6
>>> Removing node 6ee8e6e776c5c96fead4ed430a2ae206c67e5285 from cluster 127.0.0.1:8000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
$ redis-cli -p 8000 CLUSTER NODES | wc -l
6
$ redis-cli -p 8006 CLUSTER NODES
6ee8e6e776c5c96fead4ed430a2ae206c67e5285 127.0.0.1:8006@18006 myself,master - 0 0 7 connected
```

The removed node still runs, alone and empty. Stop it with `SHUTDOWN NOSAVE` and
delete its directory, or reuse it.

To do it by hand, send [`CLUSTER FORGET`](https://docs.hotkv.com/commands/cluster-forget/index.md) with the node's ID to
every other node within 60 seconds (`cluster-blacklist-ttl`): during those 60 seconds a
node that forgot it ignores what the others say about it, so gossip cannot bring it
back, and after that it can. A replica cannot forget its own primary. Then reset the
removed node with `CLUSTER RESET HARD` so it forgets the cluster too (a primary must
hold no keys for that):

```
127.0.0.1:8000> CLUSTER MEET 127.0.0.1 8006
OK
127.0.0.1:8000> CLUSTER FORGET f4b848b7bdff9b9354a046cd1483a80f6bc40298
OK
127.0.0.1:8001> CLUSTER FORGET f4b848b7bdff9b9354a046cd1483a80f6bc40298
OK
...
127.0.0.1:8006> CLUSTER RESET HARD
OK
$ redis-cli -p 8006 CLUSTER NODES | cut -d" " -f2,3,8,9
127.0.0.1:8006@18006 myself,master connected
```

## Clear a slot or a range

[`CLUSTER FLUSHSLOT`](https://docs.hotkv.com/commands/cluster-flushslot/index.md) deletes the keys of one slot, and
[`SFLUSH`](https://docs.hotkv.com/commands/sflush/index.md) the slots of the ranges you give, without touching
the rest of the node. Use them to drop a slot's data, not to move it.

## Next steps

- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md) for what happens
  when a node fails during a move.
- [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md) for the checks to run before and
  after.
- [AI engines in a cluster](https://docs.hotkv.com/clustering/ai-engines/index.md) for how AI data moves with a
  slot.
- [Cluster commands](https://docs.hotkv.com/commands/cluster/index.md): every `CLUSTER` subcommand.