Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Cluster quick start

Last updated
Not on HotKV CloudOn-Premise: no license needed

In a few minutes you will run three primaries and three replicas on one machine, write data through them, stop a primary and see a replica take its place. Everything runs on 127.0.0.1 with ports 7100 to 7105, so you can copy each block as it is. This is a laptop exercise, not a deployment: for real hosts continue with Deploy on VMs and cloud.

What you need

Section titled “What you need”
  • hotkv, installed as described in Install HotKV. Cluster mode needs no license.
  • redis-cli or valkey-cli, to create the cluster. hotkv-cli connects to a cluster node but cannot create a cluster or follow redirects (you will see this at the end). On Debian and Ubuntu, sudo apt install redis-tools installs redis-cli.
  • Ports 7100 to 7105 and 17100 to 17105 free on the machine. Each node listens on its client port and on a second port, the cluster bus, which is the client port plus 10000.

1. Start six nodes

Section titled “1. Start six nodes”

A node is a normal hotkv process started with cluster-enabled yes and its own data directory. The directory holds the node’s data files and its nodes.conf, the file where the node keeps what it knows about the cluster, so two nodes must never share one.

Terminal window
mkdir cluster-demo && cd cluster-demo
for port in 7100 7101 7102 7103 7104 7105; do
  mkdir $port
  hotkv --port $port --bind 127.0.0.1 --cluster-enabled yes \
    --cluster-node-timeout 5000 --dir "$PWD/$port" \
    --logfile hotkv.log --pidfile "$PWD/$port/hotkv.pid" --daemonize yes
done

--cluster-node-timeout 5000 shortens the time a node may be silent before the others call it failed, from 15000 ms to 5 seconds, so the failover below does not keep you waiting. Leave it out for the default. A relative --logfile is placed inside the --dir.

Every setting you can pass as --setting value can also go in a configuration file. The file for the first node is:

port 7100
bind 127.0.0.1
cluster-enabled yes
cluster-node-timeout 5000
dir /path/to/cluster-demo/7100
logfile hotkv.log

Start it with hotkv /path/to/hotkv.conf. The same settings in hotkv.toml are listed on Cluster configuration.

2. Create the cluster

Section titled “2. Create the cluster”

The six nodes know nothing about each other yet. redis-cli --cluster create introduces them, splits the 16384 hash slots over the first three as primaries and makes the other three their replicas:

Terminal window
redis-cli --cluster create 127.0.0.1:7100 127.0.0.1:7101 127.0.0.1:7102 \
  127.0.0.1:7103 127.0.0.1:7104 127.0.0.1:7105 --cluster-replicas 1 --cluster-yes
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:7104 to 127.0.0.1:7100
Adding replica 127.0.0.1:7105 to 127.0.0.1:7101
Adding replica 127.0.0.1:7103 to 127.0.0.1:7102
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100
   slots:[0-5460] (5461 slots) master
M: eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101
   slots:[5461-10922] (5462 slots) master
M: 7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102
   slots:[10923-16383] (5461 slots) master
S: 08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103
   replicates 7180e0fb64faa975871bba28cbaea245cf4a4b4e
S: 0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104
   replicates f0855ce705ec0494ae7899a06104c3326773fd5b
S: ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105
   replicates eed6ca548c2e5153ad85e3de6b0bb1502060b63d
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:7100)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

--cluster-replicas 1 asks for one replica per primary. --cluster-yes accepts the proposed layout without asking. The warning about replicas on the same host is expected here, because every node is on one machine. On real hosts, redis-cli tries to put a replica on a different host than its primary.

Which replica follows which primary is chosen at random, so your node IDs and your pairs will differ from the ones shown on this page. Read them from your own output.

Right after the nodes start, a primary that owns slots reports its cluster state as fail for a few seconds before it accepts writes. Wait until it reports ok:

Terminal window
until redis-cli -p 7100 CLUSTER INFO | grep -q cluster_state:ok; do sleep 1; done
redis-cli -p 7100 CLUSTER INFO | head -n 2
cluster_enabled:1
cluster_state:ok

3. Look at the cluster

Section titled “3. Look at the cluster”

Connect to any node with redis-cli -p 7100. CLUSTER INFO gives the health of the cluster as this node sees it:

127.0.0.1:7100> CLUSTER INFO
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_nodes_pfail:0
cluster_nodes_fail:0
cluster_voting_nodes_pfail:0
cluster_voting_nodes_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
...

cluster_state:ok with all 16384 slots assigned means every slot has a working owner. cluster_size counts the primaries that own slots. CLUSTER NODES lists every node on one line: its ID, its address as ip:port@bus-port, its flags, the ID of its primary (- for a primary), the epoch, the link state and the slots it owns:

127.0.0.1:7100> CLUSTER NODES
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000742057 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000740856 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 slave 7180e0fb64faa975871bba28cbaea245cf4a4b4e 0 1780000741857 4 connected
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 master - 0 1780000742057 3 connected 10923-16383
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000742358 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460

CLUSTER SHARDS groups the same information by shard, a primary together with its replicas (the first shard is shown):

127.0.0.1:7100> CLUSTER SHARDS
1) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "f0855ce705ec0494ae7899a06104c3326773fd5b"
          3) "port"
          4) (integer) 7100
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 0
         13) "health"
         14) "online"
      2)  1) "id"
          2) "0c1de9328b71bf5f0c837739a873a2e60e7895fb"
          3) "port"
          4) (integer) 7104
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 1
         13) "health"
         14) "online"
   5) "id"
   6) "04ab5816ec63fc046ae0595db37d376389e5f6bd"
...

4. Write and read data

Section titled “4. Write and read data”

Every key belongs to one of the 16384 slots, and each slot belongs to one primary. A node that is asked for a key it does not own answers with a MOVED redirect that names the owner. redis-cli -c follows redirects for you, and prints a line each time it does. Start it with redis-cli -c -p 7100:

127.0.0.1:7100> SET greeting hello
-> Redirected to slot [12714] located at 127.0.0.1:7102
OK
127.0.0.1:7102> GET greeting
"hello"
127.0.0.1:7102> SET user:1 Ada
-> Redirected to slot [10778] located at 127.0.0.1:7101
OK
127.0.0.1:7101> SET user:2 Grace
OK

A command that names several keys works only when all of them are in the same slot. Two different keys usually are not, even when the same node owns both:

127.0.0.1:7101> MSET user:1 Ada user:2 Grace
(error) CROSSSLOT Keys in request don't hash to the same slot

When keys must be used together, put a hash tag in them. Only the part between the first { and the next } is hashed, so keys that share a tag share a slot:

127.0.0.1:7101> MSET {user:1}:name Ada {user:1}:city Oslo
OK
127.0.0.1:7101> MGET {user:1}:name {user:1}:city
1) "Ada"
2) "Oslo"
127.0.0.1:7101> CLUSTER KEYSLOT user:1
(integer) 10778
127.0.0.1:7101> CLUSTER KEYSLOT {user:1}:name
(integer) 10778

CLUSTER KEYSLOT tells you which slot a key hashes to. See Slots and redirects for every reply a cluster can give and Keys and commands for the multi-key rules.

5. What hotkv-cli does

Section titled “5. What hotkv-cli does”

hotkv-cli -p 7100 talks to the one node you point it at. It does not follow a redirect, so a key owned by another node comes back as an error you have to act on yourself:

127.0.0.1:7100> GET greeting
(error) MOVED 12714 127.0.0.1:7102
127.0.0.1:7100> CLUSTER KEYSLOT greeting
(integer) 12714

Use redis-cli -c, valkey-cli -c or a cluster-aware client library for applications; see Clients.

6. Stop a primary

Section titled “6. Stop a primary”

Node 7102 owns the slot of greeting. Ask it for its role to see its replica. Open the session with redis-cli -p 7102; in this transcript the replica is node 7103, and yours may be another one:

127.0.0.1:7102> ROLE
1) "master"
2) (integer) 38
3) 1) 1) "127.0.0.1"
      2) "7103"
      3) "38"

Stop the node by its process ID, wait a little longer than the node timeout, and list the nodes:

Terminal window
kill $(cat 7102/hotkv.pid)
sleep 12
redis-cli -p 7100 CLUSTER NODES
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000756876 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000756375 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 master - 0 1780000755874 7 connected 10923-16383
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 master,fail - 1780000746863 1780000744560 3 disconnected
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000756876 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460

Node 7102 is flagged fail, and its replica 7103 is now a primary that owns slots 10923 to 16383. Nobody ran a command to make that happen: the replica noticed that its primary had stopped, asked the other primaries for their votes and, with a majority, took over. Your data is still there, and writes work:

127.0.0.1:7100> GET greeting
-> Redirected to slot [12714] located at 127.0.0.1:7103
"hello"
127.0.0.1:7103> SET greeting "hello again"
OK

Now start node 7102 again, with the same command and the same directory, and look at the roles:

Terminal window
hotkv --port 7102 --bind 127.0.0.1 --cluster-enabled yes \
  --cluster-node-timeout 5000 --dir "$PWD/7102" \
  --logfile hotkv.log --pidfile "$PWD/7102/hotkv.pid" --daemonize yes
sleep 5
redis-cli -p 7100 CLUSTER NODES
ef226c76763b3596028ba37be7319c955c673c36 127.0.0.1:7105@17105 slave eed6ca548c2e5153ad85e3de6b0bb1502060b63d 0 1780000760894 6 connected
0c1de9328b71bf5f0c837739a873a2e60e7895fb 127.0.0.1:7104@17104 slave f0855ce705ec0494ae7899a06104c3326773fd5b 0 1780000761394 5 connected
08b0eab150eca8a36256883601bd48b2cbdd31d3 127.0.0.1:7103@17103 master - 0 1780000762896 7 connected 10923-16383
7180e0fb64faa975871bba28cbaea245cf4a4b4e 127.0.0.1:7102@17102 slave 08b0eab150eca8a36256883601bd48b2cbdd31d3 0 1780000762396 3 connected
eed6ca548c2e5153ad85e3de6b0bb1502060b63d 127.0.0.1:7101@17101 master - 0 1780000761895 2 connected 5461-10922
f0855ce705ec0494ae7899a06104c3326773fd5b 127.0.0.1:7100@17100 myself,master - 0 0 1 connected 0-5460

The node found its old identity in its nodes.conf, learned that 7103 had taken over, and came back as a replica of 7103. A node that comes back never takes its slots back by itself. Failover, how many replicas to run and what is lost when a primary dies are covered in Replicas and failover.

7. Clean up

Section titled “7. Clean up”
Terminal window
for port in 7100 7101 7102 7103 7104 7105; do
  kill $(cat $port/hotkv.pid)
done
sleep 2
cd .. && rm -rf cluster-demo

Stopping a node with kill (SIGTERM) saves its data first when save points are configured. Deleting the directory removes the data and each node’s nodes.conf.

Next steps

Section titled “Next steps”