# Cluster in Docker Compose

> Run a six-node HotKV cluster from one Compose file, create it with a one-shot init service, reach it from the host, and test failover, reshard and restarts.

URL: https://docs.hotkv.com/clustering/docker-compose/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

This page builds a cluster of three primaries and three replicas in Docker
Compose: one `compose.yaml`, one small init script that creates the cluster,
a password, a data volume per node, and ports published on `127.0.0.1` only.
It is a way to run a real multi-node cluster on a laptop or a CI machine
without installing anything but Docker. It is one machine, so it is a good
place to learn, develop and test failover, and not a production layout: a
production cluster needs its nodes on separate hosts
([Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md)).

## What you need

- Docker with Compose v2 (`docker compose`).
- The HotKV image, called `hotkv` here as on
  [Docker and Kubernetes](https://docs.hotkv.com/operate/docker-and-kubernetes/index.md): use the name and
  tag you were given.
- `redis-cli` or `valkey-cli` on the host, to look at the cluster from
  outside. The init service needs nothing else: it uses the `hotkv-cli` that
  is in the image.
- Free ports 7000 to 7005 on `127.0.0.1`. Change them in the file if they
  are taken. The cluster bus and replication use the container network and
  are not published.

## The files

Make a directory and put three files in it. The first holds the password, in
the `.env` file that Compose reads. Keep it out of version control:

```bash
printf "HOTKV_PASSWORD=%s\n" "$(openssl rand -hex 16)" > .env
```

`compose.yaml` runs six identical nodes that differ only in their names and
the port they publish, and an `init` service that runs once:

```yaml title="compose.yaml"
name: hotkv-cluster

x-node: &node
  image: hotkv
  restart: unless-stopped
  stop_grace_period: 2m
  environment:
    HOTKVCLI_AUTH: ${HOTKV_PASSWORD:?set HOTKV_PASSWORD in the .env file}

services:
  node1:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node1",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7000"]
    ports: ["127.0.0.1:7000:6379"]
    volumes: ["node1-data:/data"]
  node2:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node2",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7001"]
    ports: ["127.0.0.1:7001:6379"]
    volumes: ["node2-data:/data"]
  node3:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node3",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7002"]
    ports: ["127.0.0.1:7002:6379"]
    volumes: ["node3-data:/data"]
  node4:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node4",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7003"]
    ports: ["127.0.0.1:7003:6379"]
    volumes: ["node4-data:/data"]
  node5:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node5",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7004"]
    ports: ["127.0.0.1:7004:6379"]
    volumes: ["node5-data:/data"]
  node6:
    <<: *node
    command: ["--cluster-enabled", "yes", "--requirepass", "${HOTKV_PASSWORD}", "--masterauth", "${HOTKV_PASSWORD}",
              "--appendonly", "yes", "--shutdown-on-sigterm", "failover",
              "--cluster-announce-ip", "node6",
              "--cluster-announce-client-ipv4", "127.0.0.1", "--cluster-announce-client-port", "7005"]
    ports: ["127.0.0.1:7005:6379"]
    volumes: ["node6-data:/data"]

  init:
    <<: *node
    restart: "no"
    entrypoint: ["/bin/bash", "/init.sh"]
    command: []
    depends_on: [node1, node2, node3, node4, node5, node6]
    volumes:
      - ./init.sh:/init.sh:ro

volumes:
  node1-data:
  node2-data:
  node3-data:
  node4-data:
  node5-data:
  node6-data:
```

`init.sh` waits for the nodes, introduces them to each other, gives the three
primaries a third of the 16384 slots each, attaches a replica to each primary
and waits for the cluster to report `cluster_state:ok`. It is plain
[`CLUSTER MEET`](https://docs.hotkv.com/commands/cluster-meet/index.md),
[`CLUSTER ADDSLOTSRANGE`](https://docs.hotkv.com/commands/cluster-addslotsrange/index.md) and
[`CLUSTER REPLICATE`](https://docs.hotkv.com/commands/cluster-replicate/index.md), so no other tool or image
is needed. Run again against a cluster that exists, it only says so:

```bash title="init.sh"
set -eu
cli() { hotkv-cli -h "$1" "${@:2}"; }

for n in node1 node2 node3 node4 node5 node6; do
  until cli "$n" ping >/dev/null 2>&1; do sleep 1; done
done

if cli node1 cluster info | grep -q 'cluster_state:ok'; then
  echo "The cluster already exists."
  exit 0
fi

# Introduce every node to node1, which tells the others.
for n in node2 node3 node4 node5 node6; do
  cli node1 cluster meet "$(getent hosts "$n" | cut -d' ' -f1)" 6379
done
until [ "$(cli node1 cluster nodes | grep -c .)" = 6 ]; do sleep 1; done

# Three primaries share the 16384 slots.
cli node1 cluster addslotsrange 0 5460
cli node2 cluster addslotsrange 5461 10922
cli node3 cluster addslotsrange 10923 16383

# The other three nodes become their replicas.
for pair in node4:node1 node5:node2 node6:node3; do
  replica=${pair%%:*}; primary=${pair##*:}
  until cli "$replica" cluster nodes | grep -q "$(cli "$primary" cluster myid)"; do sleep 1; done
  cli "$replica" cluster replicate "$(cli "$primary" cluster myid)"
done

for n in node1 node2 node3 node4 node5 node6; do
  until cli "$n" cluster info | grep -q 'cluster_state:ok'; do sleep 1; done
done
echo "The cluster is ready."
```

## Start the cluster

```bash
docker compose up -d
```

```text
 Network hotkv-cluster_default  Creating
 Network hotkv-cluster_default  Created
 ...
 Container hotkv-cluster-init-1  Starting
 Container hotkv-cluster-init-1  Started
```

The `init` container exits when the cluster is ready, a few seconds later.
Its log says so, and `docker compose ps` lists the six nodes (the status
changes from `health: starting` to `healthy` within half a minute, as the
image's own health check pings each node):

```bash
docker compose logs init | tail -3
docker compose ps --format "table {{.Service}}\t{{.Status}}\t{{.Ports}}"
```

```text
init-1  | OK
init-1  | OK
init-1  | The cluster is ready.
SERVICE   STATUS                            PORTS
node1     Up 6 seconds (health: starting)   127.0.0.1:7000->6379/tcp
node2     Up 6 seconds (health: starting)   127.0.0.1:7001->6379/tcp
node3     Up 6 seconds (health: starting)   127.0.0.1:7002->6379/tcp
node4     Up 6 seconds (health: starting)   127.0.0.1:7003->6379/tcp
node5     Up 6 seconds (health: starting)   127.0.0.1:7004->6379/tcp
node6     Up 6 seconds (health: starting)   127.0.0.1:7005->6379/tcp
```

## Check it from the host

Give `redis-cli` the password through its environment, then connect with
`-c` so that it follows redirects. The nodes tell clients the address they
can reach from the host, `127.0.0.1` and the published port:

```bash
export REDISCLI_AUTH=$(grep ^HOTKV_PASSWORD .env | cut -d= -f2)
redis-cli -c -p 7000
```

```text
127.0.0.1:7000> SET user:1000 ada
OK
127.0.0.1:7000> SET city Lisbon
-> Redirected to slot [11479] located at 127.0.0.1:7002
OK
127.0.0.1:7002> GET city
"Lisbon"
127.0.0.1:7002> CLUSTER KEYSLOT city
(integer) 11479
```

```bash
redis-cli -p 7000 CLUSTER NODES | awk '{print $2, $3, $9}' | sort
redis-cli --cluster check 127.0.0.1:7000 | tail -4
```

```text
127.0.0.1:7000@16379 myself,master 0-5460
127.0.0.1:7001@16379 master 5461-10922
127.0.0.1:7002@16379 master 10923-16383
127.0.0.1:7003@16379 slave
127.0.0.1:7004@16379 slave
127.0.0.1:7005@16379 slave
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

(The number after `@` is the cluster bus port inside the container.)

## What the settings do

| Setting | Why |
| --- | --- |
| `--requirepass`, `--masterauth` | One password for clients and for the nodes among themselves: replicas log in with it to follow their primary, and an atomic slot migration logs in with `masterauth` to the node it moves slots to. Without `--masterauth` that migration fails with `NOAUTH`. |
| `--appendonly yes` and the `/data` volume | Each node keeps its data and its `nodes.conf` in its own volume, so a restart finds the cluster as it was. |
| `--cluster-announce-ip nodeN` | The nodes reach each other by service name, which Docker resolves to the container's address when they connect. A container that comes back with another address is found again. |
| `--cluster-announce-client-ipv4`, `--cluster-announce-client-port` | What clients are told: `127.0.0.1` and the port you published. Inside a container `127.0.0.1` is the container itself, so only clients on the host can use these addresses. |
| `--shutdown-on-sigterm failover` | `docker compose stop` sends SIGTERM. With this setting a primary hands its slots to a replica before it exits, so a planned stop causes no outage. |
| `restart: unless-stopped`, `stop_grace_period: 2m` | A node that dies comes back, and a stop gives a large data set time to be saved. |
| `ports: ["127.0.0.1:7000:6379"]` | Publishes the client port on the loopback address only. The bus port (16379) stays inside the network. |

The two `cluster-announce-client` settings are what make the cluster usable
from the host. They are also why the host cannot use every tool: see
[Resharding](#resharding) below.

### Applications inside the Compose project

Clients that run as services in the same project reach the nodes by name, so
they need the names and not `127.0.0.1`. Remove the two
`--cluster-announce-client-*` settings from every node and the nodes hand out
what they announce, `node3:6379` and so on. With the same file minus those two
settings, `hotkv-cli` run inside a container shows the redirect, and a
cluster-aware Python client run in another container on the same network
connects and finds all six nodes:

```bash
docker compose exec node1 hotkv-cli get foo
docker compose exec node1 hotkv-cli cluster nodes | awk '{print $2, $3, $9}' | sort
```

```text
MOVED 12182 node3:6379
node1:6379@16379 myself,master 0-5460
node2:6379@16379 master 5461-10922
node3:6379@16379 master 10923-16383
node4:6379@16379 slave
node5:6379@16379 slave
node6:6379@16379 slave
```

```python title="client.py"
import os
from redis.cluster import RedisCluster, ClusterNode

rc = RedisCluster(startup_nodes=[ClusterNode("node1", 6379)], password=os.environ["HOTKV_PASSWORD"], decode_responses=True)
rc.set("foo", "bar")
print(rc.get("foo"))
print(sorted(f"{n.host}:{n.port}" for n in rc.get_nodes()))
```

```bash
docker run --rm --network hotkv-cluster_default --env-file .env -v "$PWD/client.py:/client.py:ro" python:3-slim sh -c "pip install -q --root-user-action=ignore redis 2>&1 | tail -1; python /client.py"
```

```text
bar
['node1:6379', 'node2:6379', 'node3:6379', 'node4:6379', 'node5:6379', 'node6:6379']
```

A cluster gives each node one endpoint. If you need both kinds of client, run
your tools inside the network, or announce addresses that both can reach, as
described in [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md).

## Resharding

Moving slots needs the nodes to reach each other, and in this layout that
works with the atomic slot migration only. `valkey-cli --cluster reshard`
with `--cluster-use-atomic-slot-migration` moved slot 0, which holds a key, to
the other node, and [`CLUSTER MIGRATESLOTS`](https://docs.hotkv.com/commands/cluster-migrateslots/index.md),
sent to the node that owns the slot and naming the node that is to get it,
moved it back. The
`redis-cli --cluster reshard` that moves keys with `MIGRATE` does not work
here: it names the nodes by the addresses clients are given, and the source
node, inside its container, cannot connect to `127.0.0.1:7001`. It stops with
`IOERR` and leaves the slot open on both nodes; `CLUSTER SETSLOT 0 STABLE` on
both closes it. (`redis-cli` 8.4 and later, which asks the target to import,
does not work in this layout either.)

```bash
SRC=$(redis-cli -p 7000 CLUSTER MYID); DST=$(redis-cli -p 7001 CLUSTER MYID)
redis-cli -c -p 7000 SET 06S x
redis-cli --cluster reshard 127.0.0.1:7000 --cluster-from $SRC --cluster-to $DST --cluster-slots 1 --cluster-yes 2>&1 | tail -4
redis-cli -p 7000 CLUSTER SETSLOT 0 STABLE; redis-cli -p 7001 CLUSTER SETSLOT 0 STABLE
export VALKEYCLI_AUTH=$REDISCLI_AUTH; valkey-cli --cluster reshard 127.0.0.1:7000 --cluster-from $SRC --cluster-to $DST --cluster-slots 1 --cluster-yes --cluster-use-atomic-slot-migration 2>&1 | tail -3
redis-cli -c -p 7000 GET 06S
redis-cli -p 7001 CLUSTER MIGRATESLOTS SLOTSRANGE 0 0 NODE $SRC
redis-cli -c -p 7000 GET 06S
```

```text
OK
Node 127.0.0.1:7000 replied with error:
IOERR error or timeout connecting to the specified instance

clusterManagerMoveSlot failed: IOERR error or timeout connecting to the specified instance
OK
OK
  Resharding plan:
    Moving slot range 0-0 from f15f91672565e9d0a623c91a55a9923c6326ef92
Moving slot range 0-0 from 127.0.0.1:7000 to 127.0.0.1:7001 via atomic slot migration.#
-> Redirected to slot [0] located at 127.0.0.1:7001
"x"
OK
"x"
```

(`06S` is a key name in slot 0. The first `GET` is redirected to `7001`
because the slot had moved there; after `MIGRATESLOTS` it is back on `7000`.
The `OK` lines are the replies of the commands that print one.)

## Fail a primary

A planned stop is the common case: `docker compose stop` sends SIGTERM, and
because of `--shutdown-on-sigterm failover` the replica of the stopped
primary takes over at once. Stop `node1`, which serves slots 0 to 5460, and
read and write through the cluster:

```bash
docker compose stop node1
redis-cli -p 7001 CLUSTER NODES | awk '{print $2, $3, $9}' | sort
redis-cli -c -p 7001
```

```text
 Container hotkv-cluster-node1-1  Stopping
 Container hotkv-cluster-node1-1  Stopped
127.0.0.1:7000@16379 master
127.0.0.1:7001@16379 myself,master 5461-10922
127.0.0.1:7002@16379 master 10923-16383
127.0.0.1:7003@16379 master 0-5460
127.0.0.1:7004@16379 slave
127.0.0.1:7005@16379 slave
127.0.0.1:7001> GET user:1000
-> Redirected to slot [1649] located at 127.0.0.1:7003
"ada"
127.0.0.1:7003> SET user:1000 grace
OK
```

`node4` (port 7003) serves slots 0 to 5460 at once after the
stop. Start `node1` again and it rejoins as a replica of `node4`:

```bash
docker compose start node1
sleep 5
redis-cli -p 7001 CLUSTER NODES | awk '{print $2, $3, $9}' | sort
```

```text
 Container hotkv-cluster-node1-1  Starting
 Container hotkv-cluster-node1-1  Started
127.0.0.1:7000@16379 slave
127.0.0.1:7001@16379 myself,master 5461-10922
127.0.0.1:7002@16379 master 10923-16383
127.0.0.1:7003@16379 master 0-5460
127.0.0.1:7004@16379 slave
127.0.0.1:7005@16379 slave
```

A crash is different: nobody tells the cluster. `docker compose kill` sends
SIGKILL, and the replica waits for the other nodes to agree that the primary
is gone, which takes the node timeout (15 seconds by default) plus the
election. Kill `node2`, serving slots 5461 to 10922, and poll `CLUSTER NODES`
through a node that is still up, here `node1`, until its replica `node5` is the
primary:

```bash
docker compose kill node2
until redis-cli -p 7000 CLUSTER NODES | grep -q '127.0.0.1:7004@16379 master'; do sleep 0.5; done
echo "node5 is the primary"
```

```text
 Container hotkv-cluster-node2-1  Killing
 Container hotkv-cluster-node2-1  Killed
node5 is the primary
```

Writes to those slots fail until then. `docker compose start node2` brings the
old primary back as a replica.

## Restart and keep the data

`docker compose down` removes the containers and the network but keeps the
volumes, and with them each node's data and `nodes.conf`. Starting again
brings the cluster back with the same node IDs and data, even when the
containers get other addresses, because the nodes find each other by name.
To prove it, start the nodes in reverse order, which changes the addresses:

```bash
for n in node1 node2 node3; do docker inspect -f "$n {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" hotkv-cluster-$n-1; done
docker compose down
for n in node6 node5 node4 node3 node2 node1; do docker compose up -d $n 2>/dev/null; done
for n in node1 node2 node3; do docker inspect -f "$n {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" hotkv-cluster-$n-1; done
redis-cli -p 7000 CLUSTER INFO | head -3
redis-cli -c -p 7000
```

```text
node1 172.18.0.4
node2 172.18.0.5
node3 172.18.0.7
node1 172.18.0.7
node2 172.18.0.6
node3 172.18.0.5
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
127.0.0.1:7000> GET user:1000
-> Redirected to slot [1649] located at 127.0.0.1:7003
"grace"
127.0.0.1:7003> GET city
-> Redirected to slot [11479] located at 127.0.0.1:7002
"Lisbon"
```

The `init` service did not run again, because `up -d <node>` starts only the
nodes you name. If you run `docker compose up -d` for the whole project
instead, `init` runs, finds `cluster_state:ok` and prints
`The cluster already exists.`.

## Tear down

```bash
docker compose down -v
```

`-v` also deletes the volumes, and with them the data. Leave it out to keep
the cluster for next time.

## Before you rely on it

- All six nodes share one machine, one disk and one Docker daemon, so they
  fail together. For availability, put the nodes on different hosts.
- The password is in the `.env` file and in each container's command line.
  Keep the file private and the published ports on `127.0.0.1`.

## Next steps

- [Run a cluster on Kubernetes](https://docs.hotkv.com/clustering/kubernetes/index.md) for the same layout
  as a StatefulSet.
- [Cluster clients](https://docs.hotkv.com/clustering/clients/index.md): connect your application.
- [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md): the production layout.