# Cluster quick start

> Start a six-node HotKV cluster on one machine, read and write through it, then stop a primary and watch its replica take over.

URL: https://docs.hotkv.com/clustering/quick-start/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

In a few minutes you will run three primaries and three replicas on one
machine, write data through them, stop a primary and see a replica take its
place. Everything runs on `127.0.0.1` with ports 7100 to 7105, so you can
copy each block as it is. This is a laptop exercise, not a deployment: for
real hosts continue with [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md).

## What you need

- `hotkv`, installed as described in [Install HotKV](https://docs.hotkv.com/get-started/install/index.md).
  Cluster mode needs no license.
- `redis-cli` or `valkey-cli`, to create the cluster. [`hotkv-cli`](https://docs.hotkv.com/get-started/hotkv-cli/index.md)
  connects to a cluster node but cannot create a cluster or follow
  redirects (you will see this at the end). On Debian and Ubuntu,
  `sudo apt install redis-tools` installs `redis-cli`.
- Ports 7100 to 7105 and 17100 to 17105 free on the machine. Each node
  listens on its client port and on a second port, the cluster bus, which
  is the client port plus 10000.

## 1. Start six nodes

A node is a normal `hotkv` process started with `cluster-enabled yes` and
its own data directory. The directory holds the node's data files and its
`nodes.conf`, the file where the node keeps what it knows about the cluster,
so two nodes must never share one.

```bash
mkdir cluster-demo && cd cluster-demo
for port in 7100 7101 7102 7103 7104 7105; do
  mkdir $port
  hotkv --port $port --bind 127.0.0.1 --cluster-enabled yes \
    --cluster-node-timeout 5000 --dir "$PWD/$port" \
    --logfile hotkv.log --pidfile "$PWD/$port/hotkv.pid" --daemonize yes
done
```

`--cluster-node-timeout 5000` shortens the time a node may be silent before
the others call it failed, from 15000 ms to 5 seconds, so the failover
below does not keep you waiting. Leave it out for the default. A relative
`--logfile` is placed inside the `--dir`.

Every setting you can pass as `--setting value` can also go in a
configuration file. The file for the first node is:

```text
port 7100
bind 127.0.0.1
cluster-enabled yes
cluster-node-timeout 5000
dir /path/to/cluster-demo/7100
logfile hotkv.log
```

Start it with `hotkv /path/to/hotkv.conf`. The same settings in
`hotkv.toml` are listed on [Cluster configuration](https://docs.hotkv.com/clustering/configuration/index.md).

## 2. Create the cluster

The six nodes know nothing about each other yet. `redis-cli --cluster
create` introduces them, splits the 16384 hash slots over the first three
as primaries and makes the other three their replicas:

```bash
redis-cli --cluster create 127.0.0.1:7100 127.0.0.1:7101 127.0.0.1:7102 \
  127.0.0.1:7103 127.0.0.1:7104 127.0.0.1:7105 --cluster-replicas 1 --cluster-yes
```

```text
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:7104 to 127.0.0.1:7100
Adding replica 127.0.0.1:7105 to 127.0.0.1:7101
Adding replica 127.0.0.1:7103 to 127.0.0.1:7102
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100
   slots:[0-5460] (5461 slots) master
M: eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101
   slots:[5461-10922] (5462 slots) master
M: 7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102
   slots:[10923-16383] (5461 slots) master
S: 08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103
   replicates 7180e0fb64faa975871bba28cbaea245cf4a4b4e
S: 0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104
   replicates f0855ce705ec0494ae7899a06104c3326773fd5b
S: ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105
   replicates eed6ca548c2e5153ad85e3de6b0bb1502060b63d
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:7100)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

`--cluster-replicas 1` asks for one replica per primary. `--cluster-yes`
accepts the proposed layout without asking. The warning about replicas on
the same host is expected here, because every node is on one machine. On
real hosts, `redis-cli` tries to put a replica on a different host than its
primary.

Which replica follows which primary is chosen at random, so your node IDs
and your pairs will differ from the ones shown on this page. Read them from
your own output.

Right after the nodes start, a primary that owns slots reports its cluster
state as `fail` for a few seconds before it accepts writes. Wait until it
reports `ok`:

```bash
until redis-cli -p 7100 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done
redis-cli -p 7100 CLUSTER INFO | head -n 2
```

```text
cluster_enabled:1
cluster_state:ok
```

## 3. Look at the cluster

Connect to any node with `redis-cli -p 7100`.
[`CLUSTER INFO`](https://docs.hotkv.com/commands/cluster-info/index.md) gives the health of the cluster as
this node sees it:

```text
127.0.0.1:7100> CLUSTER INFO
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_nodes_pfail:0
cluster_nodes_fail:0
cluster_voting_nodes_pfail:0
cluster_voting_nodes_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
...
```

`cluster_state:ok` with all 16384 slots assigned means every slot has a
working owner. `cluster_size` counts the primaries that own slots.
[`CLUSTER NODES`](https://docs.hotkv.com/commands/cluster-nodes/index.md) lists every node on one line:
its ID, its address as `ip:port@bus-port`, its flags, the ID of its primary
(`-` for a primary), the epoch, the link state and the slots it owns:

```text
127.0.0.1:7100> CLUSTER NODES
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000742057 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000740856 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 slave 7180e0fb64faa975871bba28cbaea245cf4a4b4e 0 1780000741857 4 connected
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 master - 0 1780000742057 3 connected 10923-16383
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000742358 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460
```

[`CLUSTER SHARDS`](https://docs.hotkv.com/commands/cluster-shards/index.md) groups the same information
by shard, a primary together with its replicas (the first shard is shown):

```text
127.0.0.1:7100> CLUSTER SHARDS
1) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "f0855ce705ec0494ae7899a06104c3326773fd5b"
          3) "port"
          4) (integer) 7100
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 0
         13) "health"
         14) "online"
      2)  1) "id"
          2) "0c1de9328b71bf5f0c837739a873a2e60e7895fb"
          3) "port"
          4) (integer) 7104
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 1
         13) "health"
         14) "online"
   5) "id"
   6) "04ab5816ec63fc046ae0595db37d376389e5f6bd"
...
```

## 4. Write and read data

Every key belongs to one of the 16384 slots, and each slot belongs to one
primary. A node that is asked for a key it does not own answers with a
`MOVED` redirect that names the owner. `redis-cli -c` follows redirects for
you, and prints a line each time it does. Start it with `redis-cli -c -p 7100`:

```text
127.0.0.1:7100> SET greeting hello
-> Redirected to slot [12714] located at 127.0.0.1:7102
OK
127.0.0.1:7102> GET greeting
"hello"
127.0.0.1:7102> SET user:1 Ada
-> Redirected to slot [10778] located at 127.0.0.1:7101
OK
127.0.0.1:7101> SET user:2 Grace
OK
```

A command that names several keys works only when all of them are in the
same slot. Two different keys usually are not, even when the same node owns
both:

```text
127.0.0.1:7101> MSET user:1 Ada user:2 Grace
(error) CROSSSLOT Keys in request don't hash to the same slot
```

When keys must be used together, put a hash tag in them. Only the part
between the first `{` and the next `}` is hashed, so keys that share a tag
share a slot:

```text
127.0.0.1:7101> MSET {user:1}:name Ada {user:1}:city Oslo
OK
127.0.0.1:7101> MGET {user:1}:name {user:1}:city
1) "Ada"
2) "Oslo"
127.0.0.1:7101> CLUSTER KEYSLOT user:1
(integer) 10778
127.0.0.1:7101> CLUSTER KEYSLOT {user:1}:name
(integer) 10778
```

[`CLUSTER KEYSLOT`](https://docs.hotkv.com/commands/cluster-keyslot/index.md) tells you which slot a key
hashes to. See [Slots and redirects](https://docs.hotkv.com/clustering/index.md#slots-and-redirects) for
every reply a cluster can give and
[Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md) for the multi-key
rules.

## 5. What `hotkv-cli` does

`hotkv-cli -p 7100` talks to the one node you point it at. It does not
follow a redirect, so a key owned by another node comes back as an error you
have to act on yourself:

```text
127.0.0.1:7100> GET greeting
(error) MOVED 12714 127.0.0.1:7102
127.0.0.1:7100> CLUSTER KEYSLOT greeting
(integer) 12714
```

Use `redis-cli -c`, `valkey-cli -c` or a cluster-aware client
library for applications; see [Clients](https://docs.hotkv.com/clustering/clients/index.md).

## 6. Stop a primary

Node 7102 owns the slot of `greeting`. Ask it for its role to see its
replica. Open the session with `redis-cli -p 7102`; in this transcript the
replica is node 7103, and yours may be another one:

```text
127.0.0.1:7102> ROLE
1) "master"
2) (integer) 38
3) 1) 1) "127.0.0.1"
      2) "7103"
      3) "38"
```

Stop the node by its process ID, wait a little longer than the node
timeout, and list the nodes:

```bash
kill $(cat 7102/hotkv.pid)
sleep 12
redis-cli -p 7100 CLUSTER NODES
```

```text
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000756876 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000756375 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 master - 0 1780000755874 7 connected 10923-16383
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 master,fail - 1780000746863 1780000744560 3 disconnected
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000756876 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460
```

Node 7102 is flagged `fail`, and its replica 7103 is now a primary that owns
slots 10923 to 16383. Nobody ran a command to make that happen: the replica
noticed that its primary had stopped, asked the other primaries for their
votes and, with a majority, took over. Your data is still there, and writes
work:

```text
127.0.0.1:7100> GET greeting
-> Redirected to slot [12714] located at 127.0.0.1:7103
"hello"
127.0.0.1:7103> SET greeting "hello again"
OK
```

Now start node 7102 again, with the same command and the same directory,
and look at the roles:

```bash
hotkv --port 7102 --bind 127.0.0.1 --cluster-enabled yes \
  --cluster-node-timeout 5000 --dir "$PWD/7102" \
  --logfile hotkv.log --pidfile "$PWD/7102/hotkv.pid" --daemonize yes
sleep 5
redis-cli -p 7100 CLUSTER NODES
```

```text
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000760894 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000761394 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 master - 0 1780000762896 7 connected 10923-16383
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 slave 08b0eab150eca8a36256883601bd48b2cbdd31d3 0 1780000762396 3 connected
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000761895 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460
```

The node found its old identity in its `nodes.conf`, learned that 7103 had
taken over, and came back as a replica of 7103. A node that comes back
never takes its slots back by itself. Failover, how many replicas to run
and what is lost when a primary dies are covered in
[Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md).

## 7. Clean up

```bash
for port in 7100 7101 7102 7103 7104 7105; do
  kill $(cat $port/hotkv.pid)
done
sleep 2
cd .. && rm -rf cluster-demo
```

Stopping a node with `kill` (SIGTERM) saves its data first when save points
are configured. Deleting the directory removes the data and each node's
`nodes.conf`.

## Next steps

- [Cluster overview](https://docs.hotkv.com/clustering/index.md): what a cluster is for, how many nodes
  you need and what it cannot do.
- [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md): real hosts, firewalls,
  passwords and `systemd`.
- [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md): encrypt clients and the links
  between nodes.
- [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md): what the error messages
  mean.