Skip to content

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Keys, slots and commands in a cluster

Last updated
Not on HotKV CloudOn-Premise: no license needed

A cluster spreads your keys over its primaries, and most of what is different about using HotKV in a cluster follows from that one fact: a command can only touch keys that live on one node. This page shows how a key finds its node, how to keep related keys together, what that means for multi-key commands, transactions, scripts, blocking commands, Pub/Sub and databases, and which commands answer for a single node instead of the whole cluster.

The examples use the six-node cluster of the quick start (three primaries, three replicas) on ports 7100 to 7105, with no password. In this cluster the primary on port 7100 serves slots 0 to 5460, 7101 serves 5461 to 10922 and 7102 serves 10923 to 16383. Change the ports to match your cluster.

Every key belongs to one of 16384 hash slots, the CRC16 of the key modulo 16384, and every slot is served by exactly one primary. CLUSTER KEYSLOT tells you the slot of any name. A node that gets a command for a slot it does not serve answers MOVED slot address and the client sends the command there:

127.0.0.1:7100> CLUSTER KEYSLOT user:1000
(integer) 1649
127.0.0.1:7100> CLUSTER KEYSLOT user:1001
(integer) 5712
127.0.0.1:7100> CLUSTER KEYSLOT city
(integer) 11479
127.0.0.1:7100> SET user:1000 ada
OK
127.0.0.1:7100> GET user:1000
"ada"
127.0.0.1:7100> SET city Lisbon
(error) MOVED 11479 127.0.0.1:7102

redis-cli -c follows the redirect for you, which is what a cluster-aware client library does too (see Cluster clients):

$ redis-cli -c -p 7100
127.0.0.1:7100> SET city Lisbon
-> Redirected to slot [11479] located at 127.0.0.1:7102
OK
127.0.0.1:7102> GET city
"Lisbon"

The other redirects and errors a client can meet (ASK, TRYAGAIN, CLUSTERDOWN) are explained in Slots and redirects.

When a key contains a pair of braces with at least one character between them, only the text inside the first pair is hashed. Keys that share that text land in the same slot, so on the same node:

127.0.0.1:7100> CLUSTER KEYSLOT {user:1000}.profile
(integer) 1649
127.0.0.1:7100> CLUSTER KEYSLOT {user:1000}.orders
(integer) 1649
127.0.0.1:7100> CLUSTER KEYSLOT user:1000
(integer) 1649
127.0.0.1:7100> SET {user:1000}.profile ada@example.com
OK
127.0.0.1:7100> SET {user:1000}.orders 3
OK
127.0.0.1:7100> MGET {user:1000}.profile {user:1000}.orders
1) "ada@example.com"
2) "3"

A few details that catch people out:

  • The tag is the text between the first { and the first } after it. In {{cart}}9 that is {cart, so the slot is the slot of {cart (3844), not of cart.
  • Empty braces are not a tag: order{}7 is hashed as a whole.
  • user and {user} hash to the same slot (5474), because the tag of {user} is user.
127.0.0.1:7100> CLUSTER KEYSLOT order{}7
(integer) 11423
127.0.0.1:7100> CLUSTER KEYSLOT {{cart}}9
(integer) 3844
127.0.0.1:7100> CLUSTER KEYSLOT {cart
(integer) 3844
127.0.0.1:7100> CLUSTER KEYSLOT user
(integer) 5474
127.0.0.1:7100> CLUSTER KEYSLOT {user}
(integer) 5474

A command that names more than one key needs all of them in one slot. Otherwise the node replies CROSSSLOT and does nothing. MSET, DEL, RENAME, SUNION, LMOVE and the other multi-key commands follow this rule, and hash tags are how you satisfy it:

127.0.0.1:7100> MSET user:1000 a user:1001 b
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> DEL user:1000 user:1001
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> RENAME user:1000 user:1001
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> SUNION s1 s2
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> LMOVE l1 l2 LEFT RIGHT
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> RENAME {user:1000}.orders {user:1000}.orders.old
OK
127.0.0.1:7100> SUNIONSTORE {s}3 {s}1 {s}2
(integer) 0
127.0.0.1:7100> EXISTS {k}a {k}b
-> Redirected to slot [7629] located at 127.0.0.1:7101
(integer) 0

The last command shows the other half of the rule: keys that share a slot are one request, and it goes to the node that serves the slot.

MULTI and EXEC run on one node, so all the keys of all the queued commands must be in one slot. A command for a slot another node serves is refused when you queue it and the transaction is discarded at EXEC. Two keys on the same node but in different slots are queued, and EXEC then answers CROSSSLOT:

127.0.0.1:7100> MULTI
OK
127.0.0.1:7100(TX)> SET {user:1000}.a 1
QUEUED
127.0.0.1:7100(TX)> SET {user:1000}.b 2
QUEUED
127.0.0.1:7100(TX)> EXEC
1) OK
2) OK
127.0.0.1:7100> MULTI
OK
127.0.0.1:7100(TX)> SET user:1000 x
QUEUED
127.0.0.1:7100(TX)> SET foo y
(error) MOVED 12182 127.0.0.1:7102
127.0.0.1:7100(TX)> EXEC
(error) EXECABORT Transaction discarded because of previous errors.
127.0.0.1:7100> MULTI
OK
127.0.0.1:7100(TX)> SET user:1000 x
QUEUED
127.0.0.1:7100(TX)> SET b y
QUEUED
127.0.0.1:7100(TX)> EXEC
(error) CROSSSLOT Keys in request don't hash to the same slot

(redis-cli -c cannot follow a redirect in the middle of a transaction, so these were typed on a plain connection to the node.) Give the keys of a transaction a common hash tag.

EVAL, EVALSHA and FCALL are routed by their declared keys, which must be in one slot. What the script does with other keys through redis.call() depends on how it starts:

  • A script written the older way, without a #!lua first line, may touch keys it did not declare, in any slot, as long as the node serves that slot.
  • A script that starts with #!lua is held to one slot: every key it touches must hash to the slot of its declared keys (or, with none declared, of the first key it uses).
  • #!lua flags=allow-cross-slot-keys lifts that for keys on the same node. A key on another node is always refused.
  • #!lua flags=no-cluster makes the script refuse to run in cluster mode.
127.0.0.1:7100> CLUSTER KEYSLOT b
(integer) 3300
127.0.0.1:7100> EVAL "return redis.call('GET', KEYS[1])" 1 user:1000
"ada"
127.0.0.1:7100> EVAL "return redis.call('GET', KEYS[1])" 2 user:1000 b
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> EVAL "return redis.call('GET', 'foo')" 0
(error) ERR Script attempted to access a non local key in a cluster node script: 8a834a7d18be1c3adf6cd25d34bc92a10665fb77, on @user_script:1.
127.0.0.1:7100> EVAL "redis.call('GET', 'user:1000'); return redis.call('GET', 'b')" 0
(nil)
127.0.0.1:7100> EVAL "#!lua\nredis.call('GET', 'user:1000')\nreturn redis.call('GET', 'b')" 0
(error) ERR Script attempted to access keys that do not hash to the same slot script: bf328d3a7ecefd319a7c628bb42ed595044f146b, on @user_script:3.
127.0.0.1:7100> EVAL "#!lua\nreturn redis.call('MSET', KEYS[1], 'a', 'b', 'c')" 1 user:1000
(error) ERR Command 'mset' in script attempted to access keys that don't hash to the same slot script: 533f9c0a1345c221a574433bcac806fae003f6ea, on @user_script:2.
127.0.0.1:7100> EVAL "#!lua flags=allow-cross-slot-keys\nredis.call('GET', 'user:1000')\nreturn redis.call('GET', 'b')" 0
(nil)
127.0.0.1:7100> EVAL "#!lua flags=allow-cross-slot-keys\nreturn redis.call('GET', 'foo')" 0
(error) ERR Script attempted to access a non local key in a cluster node script: c301fed4d1617a5091bfbe26f158d320eb493459, on @user_script:2.
127.0.0.1:7100> EVAL "#!lua flags=no-cluster\nreturn 1" 0
(error) ERR Can not run script on cluster, 'no-cluster' flag is set.

The script hashes in the messages are the SHA1 of the script text. Functions take the same two flags in redis.register_function:

127.0.0.1:7100> FUNCTION LOAD "#!lua name=demo\nredis.register_function{function_name='get_one', callback=function(keys, args) return redis.call('GET', keys[1]) end, flags={'no-writes'}}\nredis.register_function{function_name='local_only', callback=function(keys, args) return 1 end, flags={'no-cluster'}}"
"demo"
127.0.0.1:7100> FCALL get_one 1 user:1000
"ada"
127.0.0.1:7100> FCALL local_only 0
(error) ERR Can not run script on cluster, 'no-cluster' flag is set.
127.0.0.1:7100> FUNCTION LOAD "#!lua name=cross\nredis.register_function{function_name='both', callback=function(keys, args) redis.call('GET', 'user:1000'); return redis.call('GET', 'b') end, flags={'no-writes', 'allow-cross-slot-keys'}}\nredis.register_function{function_name='strict', callback=function(keys, args) redis.call('GET', 'user:1000'); return redis.call('GET', 'b') end, flags={'no-writes'}}"
"cross"
127.0.0.1:7100> FCALL both 0
(nil)
127.0.0.1:7100> FCALL strict 0
(error) ERR Script attempted to access keys that do not hash to the same slot script: strict, on @user_function:3.

The simplest habit: pass every key in KEYS, give them a common hash tag, and start the script with #!lua. A script or a function library is stored on the node you load it on, not on the cluster: see What cannot be clustered.

A blocking command such as BLPOP with several keys needs them in one slot, like any multi-key command. With a common tag it is routed to the node that serves them:

127.0.0.1:7100> BLPOP q1 q2 0.2
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7100> BLPOP {q}1 {q}2 0.2
-> Redirected to slot [11958] located at 127.0.0.1:7102
(nil)

PUBLISH is not tied to a slot: it is forwarded to the subscribers of the channel on every node, so a subscriber can be connected to any node. The number it returns counts only the subscribers on the node you sent it to. Sharded Pub/Sub (SPUBLISH, SSUBSCRIBE) keeps a channel on the node that serves its slot, and other nodes answer MOVED.

With a subscriber on port 7101 (SUBSCRIBE news) and one on port 7100 (SSUBSCRIBE {user:1000}news):

127.0.0.1:7100> PUBLISH news hello
(integer) 0
127.0.0.1:7100> SPUBLISH {user:1000}news hello
(integer) 1
127.0.0.1:7101> SPUBLISH {user:1000}news hello
(error) MOVED 1649 127.0.0.1:7100
$ redis-cli --no-raw -p 7101 SUBSCRIBE news
1) "subscribe"
2) "news"
3) (integer) 1
1) "message"
2) "news"
3) "hello"
$ redis-cli --no-raw -p 7100 SSUBSCRIBE {user:1000}news
1) "ssubscribe"
2) "{user:1000}news"
3) (integer) 1
1) "smessage"
2) "{user:1000}news"
3) "hello"

By default a cluster node has one database, as in Redis Cluster, and the commands that move data between databases are refused:

127.0.0.1:7100> SELECT 1
(error) ERR SELECT is not allowed in cluster mode
127.0.0.1:7100> MOVE user:1000 1
(error) ERR MOVE is not allowed in cluster mode
127.0.0.1:7100> COPY {user:1000}.a {user:1000}.b DB 1
(error) ERR Copying to another database is not allowed in cluster mode
127.0.0.1:7100> SWAPDB 0 1
(error) ERR SWAPDB is not allowed in cluster mode
127.0.0.1:7100> CONFIG GET cluster-databases
1) "cluster-databases"
2) "1"

cluster-databases gives a node 1 to 128 databases. Set it on every node when you start them; it cannot be changed with CONFIG SET. On the command line, a value outside 1 to 128 stops the start. In the configuration file it is only logged and the node runs with 1 database, so check the log. With a value above 1, SELECT, MOVE and COPY ... DB work (the key keeps its slot in every database), an index out of range is refused, and SWAPDB is still refused. This cluster was started with --cluster-databases 4:

127.0.0.1:7110> CONFIG GET cluster-databases
1) "cluster-databases"
2) "4"
127.0.0.1:7110> SELECT 1
OK
127.0.0.1:7110[1]> SET {u}k v1
-> Redirected to slot [11826] located at 127.0.0.1:7112
OK
127.0.0.1:7112[1]> SELECT 0
OK
127.0.0.1:7112> SET {u}k v0
OK
127.0.0.1:7112> MOVE {u}k 1
(integer) 0
127.0.0.1:7112> SELECT 1
OK
127.0.0.1:7112[1]> COPY {u}k {u}k2 DB 3
(integer) 1
127.0.0.1:7112[1]> SELECT 3
OK
127.0.0.1:7112[3]> GET {u}k2
"v1"
127.0.0.1:7112[3]> SWAPDB 0 1
(error) ERR SWAPDB is not allowed in cluster mode
127.0.0.1:7112[3]> SELECT 9
(error) ERR DB index is out of range

MOVE answered 0 because database 1 already held {u}k. The AI engines are not in a database: an entry is found by its name whichever database you selected (see AI engines in a cluster).

These commands answer for the node you send them to, as in Redis Cluster: DBSIZE, KEYS, SCAN, RANDOMKEY, FLUSHDB, FLUSHALL and INFO keyspace. Thirty keys written through the cluster are spread over the primaries:

$ for p in 7100 7101 7102; do echo "port $p: $(redis-cli -p $p DBSIZE)"; done
port 7100: 18
port 7101: 9
port 7102: 9
$ redis-cli -p 7100
127.0.0.1:7100> DBSIZE
(integer) 18
127.0.0.1:7100> KEYS key:1*
1) "key:19"
2) "key:15"
3) "key:11"
4) "key:18"
5) "key:14"
127.0.0.1:7100> INFO keyspace
# Keyspace
db0:keys=18,expires=0,avg_ttl=0,subexpiry=0

FLUSHDB and FLUSHALL are just as local. On a cluster that held 30 keys, emptying the node on port 7101 left the others alone:

127.0.0.1:7101> FLUSHDB
OK
127.0.0.1:7101> DBSIZE
(integer) 0
$ for p in 7100 7101 7102; do echo "port $p: $(redis-cli -p $p DBSIZE)"; done
port 7100: 13
port 7101: 0
port 7102: 8

To go through every key of the cluster, use CLUSTERSCAN. It takes a cursor like SCAN, and the command goes from node to node as the scan advances. Pass back the cursor you were given exactly as it is, and stop when it is 0. With redis-cli -c following the redirects, this script prints every key once:

#!/bin/bash
cursor=0
while :; do
reply=$(redis-cli -c -p 7100 CLUSTERSCAN "$cursor" COUNT 100)
cursor=$(printf '%s\n' "$reply" | head -n 1)
printf '%s\n' "$reply" | tail -n +2
[ "$cursor" = 0 ] && break
done
$ bash clusterscan.sh | sort -V | head -3
city
key:1
key:2
$ bash clusterscan.sh | wc -l
36
$ bash clusterscan.sh | sort | uniq -d | wc -l
0

Every key is listed once.

A replica answers MOVED for every key, even for the ones its primary serves, until the connection sends READONLY. Then it serves reads for the slots of its own primary, and still refuses writes and every other slot. READWRITE turns it off. Replication is asynchronous, so a replica can be a moment behind: WAIT on the primary tells you when a write has reached it. Port 7105 is the replica of the primary on 7102 here:

127.0.0.1:7102> SET city Lisbon
OK
127.0.0.1:7102> WAIT 1 1000
(integer) 1
127.0.0.1:7102> ROLE
1) "master"
2) (integer) 70
3) 1) 1) "127.0.0.1"
2) "7105"
3) "70"
127.0.0.1:7105> GET city
(error) MOVED 11479 127.0.0.1:7102
127.0.0.1:7105> READONLY
OK
127.0.0.1:7105> GET city
"Lisbon"
127.0.0.1:7105> SET city Porto
(error) MOVED 11479 127.0.0.1:7102
127.0.0.1:7105> GET user:1000
(error) MOVED 1649 127.0.0.1:7100
127.0.0.1:7105> READWRITE
OK
127.0.0.1:7105> GET city
(error) MOVED 11479 127.0.0.1:7102

Cluster-aware client libraries have an option that sends the READONLY for you and spreads reads over replicas: see Cluster clients.

These commands go to the node that serves the slot of their key. A key moves with its slot when you reshard. Sent to a node that does not serve the slot, a command of each family below is answered with MOVED to the slot of its key, or served by the node that owns it:

HSET h:1 f v slot 16102 -> MOVED 16102 127.0.0.1:7102
LPUSH l:1 a slot 8742 -> MOVED 8742 127.0.0.1:7101
SADD s:1 a slot 3444 -> 1
ZADD z:1 1 a slot 5093 -> 1
XADD x:1 * f v slot 15749 -> MOVED 15749 127.0.0.1:7102
GEOADD g:1 13.4 52.5 berlin slot 4823 -> 1
SETBIT b:1 1 1 slot 14631 -> MOVED 14631 127.0.0.1:7102
PFADD p:1 a slot 5156 -> 1
BF.ADD bf:1 a slot 4445 -> 1
CMS.INITBYDIM cms:1 100 5 slot 6702 -> MOVED 6702 127.0.0.1:7101
TOPK.RESERVE tk:1 3 slot 12082 -> MOVED 12082 127.0.0.1:7102
GCRA.LIMIT gc:1 60 5 slot 1768 -> 0

(Each command is sent to port 7100, which serves slots 0 to 5460.)

What Routed by
Strings, hashes (with field expiry), lists, sets, sorted sets, streams, geospatial, bitmaps, HyperLogLog, Bloom filters, Count-Min Sketch, Top-K the key
GCRA.* the key
The AI engines, for example PCACHE.*, SCACHE.*, VSIM.*, RAG.*, AMEM.*, LLMBUDGET.*, RATELIMIT.*, PROMPT.* and TS.* the name of the entry (cache namespace, index, pipeline, agent or session id, …)

The second part of the table is covered in AI engines in a cluster, which shows the routing of each family.

Some state belongs to a node, and some commands answer from one node only. The cluster does not hide any of it. The transcripts follow the table.

Per node What you see What to do
DBSIZE, KEYS, SCAN, RANDOMKEY, FLUSHDB, FLUSHALL, INFO keyspace Only the node you ask. Ask every primary, or use CLUSTERSCAN for keys.
The LIST commands of the AI engines, TS.LIST, VSIM.INFO without an index name Only the entries on that node. Ask every primary and merge the answers.
Scripts (the SCRIPT LOAD cache) Cached on the node that loaded them, not on its replicas. Load them on every primary that runs them, and send them again after a failover, when EVALSHA answers NOSCRIPT.
Function libraries (FUNCTION LOAD) On the node that loaded them and on its replicas, not on the other primaries. Load them on every primary.
ACL users (ACL SETUSER) Created on that node only, not even on its replicas. Run the ACL commands on every node, replicas included.
CONFIG, INFO The node you ask. Send them to every node.
The TAG.* index Stays on the node where it was written. See below.

Scripts, functions, ACL users and CONFIG in practice:

127.0.0.1:7100> SCRIPT LOAD "return redis.call('GET', KEYS[1])"
"d3c21d0c2b9ca22f82737626a27bcaf5d288f99f"
127.0.0.1:7102> EVALSHA d3c21d0c2b9ca22f82737626a27bcaf5d288f99f 1 foo
(error) NOSCRIPT No matching script. Please use EVAL.
127.0.0.1:7102> FCALL get_one 1 foo
(error) ERR Function not found
127.0.0.1:7100> ACL SETUSER reporter on nopass ~report:* +get
OK
127.0.0.1:7101> ACL GETUSER reporter
(nil)
127.0.0.1:7100> ROLE
1) "master"
2) (integer) 1185
3) 1) 1) "127.0.0.1"
2) "7103"
3) "1185"
127.0.0.1:7103> ACL GETUSER reporter
(nil)
127.0.0.1:7100> CONFIG SET maxmemory-policy allkeys-lru
OK
127.0.0.1:7101> CONFIG GET maxmemory-policy
1) "maxmemory-policy"
2) "noeviction"

TS.LIST and VSIM.INFO are examples of listings that answer per node. On a cluster with several time series and indexes, each primary lists its own:

$ for p in 7003 7004 7005; do echo "port $p: $(redis-cli -p $p VSIM.INFO | tr "\n" " ")"; done
port 7003: products
port 7004: {docs}:index
port 7005:
$ for p in 7003 7004 7005; do echo "port $p: $(redis-cli -p $p TS.LIST | tr "\n" " ")"; done
port 7003: sensor-2:raw
port 7004: sensor-2:1m sensor-1
port 7005: {sensor-3}:raw {sensor-3}:1m

The key tags (TAG.SET, TAG.GET, TAG.MEMBERS) are routed by key, but they do not move with a reshard: after CLUSTER MIGRATESLOTS moves the slot of {tag}k, the key is on the new node and its tags are gone. Keep labels that must follow a key in an ordinary set, which is a key and moves with its slot:

127.0.0.1:7100> SET {tag}k 1
-> Redirected to slot [8338] located at 127.0.0.1:7101
OK
127.0.0.1:7101> TAG.SET {tag}k red
(integer) 1
127.0.0.1:7101> TAG.GET {tag}k
1) "red"
127.0.0.1:7101> CLUSTER KEYSLOT {tag}k
(integer) 8338
$ redis-cli -p 7101 CLUSTER MIGRATESLOTS SLOTSRANGE 8338 8338 NODE $ID0
OK
127.0.0.1:7100> GET {tag}k
"1"
127.0.0.1:7100> TAG.GET {tag}k
(empty array)
127.0.0.1:7101> TAG.MEMBERS red
(empty array)
127.0.0.1:7100> SET {tag}k 1
-> Redirected to slot [8338] located at 127.0.0.1:7101
OK
127.0.0.1:7101> TAG.SET {tag}k red
(integer) 1
127.0.0.1:7101> DEL {tag}k
(integer) 1
127.0.0.1:7101> TAG.MEMBERS red
1) "{tag}k"

($ID0 is the node ID of the node on port 7100, from CLUSTER MYID.)