Keys, slots and commands in a cluster
A cluster spreads your keys over its primaries, and most of what is different about using HotKV in a cluster follows from that one fact: a command can only touch keys that live on one node. This page shows how a key finds its node, how to keep related keys together, what that means for multi-key commands, transactions, scripts, blocking commands, Pub/Sub and databases, and which commands answer for a single node instead of the whole cluster.
The examples use the six-node cluster of the quick start (three primaries, three replicas) on ports 7100 to 7105, with no password. In this cluster the primary on port 7100 serves slots 0 to 5460, 7101 serves 5461 to 10922 and 7102 serves 10923 to 16383. Change the ports to match your cluster.
How a key finds its node
Section titled “How a key finds its node”Every key belongs to one of 16384 hash slots, the CRC16 of the key modulo
16384, and every slot is served by exactly one primary.
CLUSTER KEYSLOT tells you the slot of any
name. A node that gets a command for a slot it does not serve answers
MOVED slot address and the client sends the command there:
127.0.0.1:7100> CLUSTER KEYSLOT user:1000(integer) 1649127.0.0.1:7100> CLUSTER KEYSLOT user:1001(integer) 5712127.0.0.1:7100> CLUSTER KEYSLOT city(integer) 11479127.0.0.1:7100> SET user:1000 adaOK127.0.0.1:7100> GET user:1000"ada"127.0.0.1:7100> SET city Lisbon(error) MOVED 11479 127.0.0.1:7102redis-cli -c follows the redirect for you, which is what a cluster-aware
client library does too (see Cluster clients):
$ redis-cli -c -p 7100127.0.0.1:7100> SET city Lisbon-> Redirected to slot [11479] located at 127.0.0.1:7102OK127.0.0.1:7102> GET city"Lisbon"The other redirects and errors a client can meet (ASK, TRYAGAIN,
CLUSTERDOWN) are explained in Slots and redirects.
Hash tags
Section titled “Hash tags”When a key contains a pair of braces with at least one character between them, only the text inside the first pair is hashed. Keys that share that text land in the same slot, so on the same node:
127.0.0.1:7100> CLUSTER KEYSLOT {user:1000}.profile(integer) 1649127.0.0.1:7100> CLUSTER KEYSLOT {user:1000}.orders(integer) 1649127.0.0.1:7100> CLUSTER KEYSLOT user:1000(integer) 1649127.0.0.1:7100> SET {user:1000}.profile ada@example.comOK127.0.0.1:7100> SET {user:1000}.orders 3OK127.0.0.1:7100> MGET {user:1000}.profile {user:1000}.orders1) "ada@example.com"2) "3"A few details that catch people out:
- The tag is the text between the first
{and the first}after it. In{{cart}}9that is{cart, so the slot is the slot of{cart(3844), not ofcart. - Empty braces are not a tag:
order{}7is hashed as a whole. userand{user}hash to the same slot (5474), because the tag of{user}isuser.
127.0.0.1:7100> CLUSTER KEYSLOT order{}7(integer) 11423127.0.0.1:7100> CLUSTER KEYSLOT {{cart}}9(integer) 3844127.0.0.1:7100> CLUSTER KEYSLOT {cart(integer) 3844127.0.0.1:7100> CLUSTER KEYSLOT user(integer) 5474127.0.0.1:7100> CLUSTER KEYSLOT {user}(integer) 5474Commands with several keys
Section titled “Commands with several keys”A command that names more than one key needs all of them in one slot.
Otherwise the node replies CROSSSLOT and does nothing. MSET, DEL,
RENAME, SUNION, LMOVE and the other multi-key commands follow this
rule, and hash tags are how you satisfy it:
127.0.0.1:7100> MSET user:1000 a user:1001 b(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> DEL user:1000 user:1001(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> RENAME user:1000 user:1001(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> SUNION s1 s2(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> LMOVE l1 l2 LEFT RIGHT(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> RENAME {user:1000}.orders {user:1000}.orders.oldOK127.0.0.1:7100> SUNIONSTORE {s}3 {s}1 {s}2(integer) 0127.0.0.1:7100> EXISTS {k}a {k}b-> Redirected to slot [7629] located at 127.0.0.1:7101(integer) 0The last command shows the other half of the rule: keys that share a slot are one request, and it goes to the node that serves the slot.
Transactions
Section titled “Transactions”MULTI and EXEC run on one node, so
all the keys of all the queued commands must be in one slot. A command for a
slot another node serves is refused when you queue it and the transaction is
discarded at EXEC. Two keys on the same node but in different slots are
queued, and EXEC then answers CROSSSLOT:
127.0.0.1:7100> MULTIOK127.0.0.1:7100(TX)> SET {user:1000}.a 1QUEUED127.0.0.1:7100(TX)> SET {user:1000}.b 2QUEUED127.0.0.1:7100(TX)> EXEC1) OK2) OK127.0.0.1:7100> MULTIOK127.0.0.1:7100(TX)> SET user:1000 xQUEUED127.0.0.1:7100(TX)> SET foo y(error) MOVED 12182 127.0.0.1:7102127.0.0.1:7100(TX)> EXEC(error) EXECABORT Transaction discarded because of previous errors.127.0.0.1:7100> MULTIOK127.0.0.1:7100(TX)> SET user:1000 xQUEUED127.0.0.1:7100(TX)> SET b yQUEUED127.0.0.1:7100(TX)> EXEC(error) CROSSSLOT Keys in request don't hash to the same slot(redis-cli -c cannot follow a redirect in the middle of a transaction, so
these were typed on a plain connection to the node.) Give the keys of a
transaction a common hash tag.
Scripts and functions
Section titled “Scripts and functions”EVAL, EVALSHA and
FCALL are routed by their declared keys, which must be in
one slot. What the script does with other keys through redis.call() depends
on how it starts:
- A script written the older way, without a
#!luafirst line, may touch keys it did not declare, in any slot, as long as the node serves that slot. - A script that starts with
#!luais held to one slot: every key it touches must hash to the slot of its declared keys (or, with none declared, of the first key it uses). #!lua flags=allow-cross-slot-keyslifts that for keys on the same node. A key on another node is always refused.#!lua flags=no-clustermakes the script refuse to run in cluster mode.
127.0.0.1:7100> CLUSTER KEYSLOT b(integer) 3300127.0.0.1:7100> EVAL "return redis.call('GET', KEYS[1])" 1 user:1000"ada"127.0.0.1:7100> EVAL "return redis.call('GET', KEYS[1])" 2 user:1000 b(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> EVAL "return redis.call('GET', 'foo')" 0(error) ERR Script attempted to access a non local key in a cluster node script: 8a834a7d18be1c3adf6cd25d34bc92a10665fb77, on @user_script:1.127.0.0.1:7100> EVAL "redis.call('GET', 'user:1000'); return redis.call('GET', 'b')" 0(nil)127.0.0.1:7100> EVAL "#!lua\nredis.call('GET', 'user:1000')\nreturn redis.call('GET', 'b')" 0(error) ERR Script attempted to access keys that do not hash to the same slot script: bf328d3a7ecefd319a7c628bb42ed595044f146b, on @user_script:3.127.0.0.1:7100> EVAL "#!lua\nreturn redis.call('MSET', KEYS[1], 'a', 'b', 'c')" 1 user:1000(error) ERR Command 'mset' in script attempted to access keys that don't hash to the same slot script: 533f9c0a1345c221a574433bcac806fae003f6ea, on @user_script:2.127.0.0.1:7100> EVAL "#!lua flags=allow-cross-slot-keys\nredis.call('GET', 'user:1000')\nreturn redis.call('GET', 'b')" 0(nil)127.0.0.1:7100> EVAL "#!lua flags=allow-cross-slot-keys\nreturn redis.call('GET', 'foo')" 0(error) ERR Script attempted to access a non local key in a cluster node script: c301fed4d1617a5091bfbe26f158d320eb493459, on @user_script:2.127.0.0.1:7100> EVAL "#!lua flags=no-cluster\nreturn 1" 0(error) ERR Can not run script on cluster, 'no-cluster' flag is set.The script hashes in the messages are the SHA1 of the script text. Functions
take the same two flags in redis.register_function:
127.0.0.1:7100> FUNCTION LOAD "#!lua name=demo\nredis.register_function{function_name='get_one', callback=function(keys, args) return redis.call('GET', keys[1]) end, flags={'no-writes'}}\nredis.register_function{function_name='local_only', callback=function(keys, args) return 1 end, flags={'no-cluster'}}""demo"127.0.0.1:7100> FCALL get_one 1 user:1000"ada"127.0.0.1:7100> FCALL local_only 0(error) ERR Can not run script on cluster, 'no-cluster' flag is set.127.0.0.1:7100> FUNCTION LOAD "#!lua name=cross\nredis.register_function{function_name='both', callback=function(keys, args) redis.call('GET', 'user:1000'); return redis.call('GET', 'b') end, flags={'no-writes', 'allow-cross-slot-keys'}}\nredis.register_function{function_name='strict', callback=function(keys, args) redis.call('GET', 'user:1000'); return redis.call('GET', 'b') end, flags={'no-writes'}}""cross"127.0.0.1:7100> FCALL both 0(nil)127.0.0.1:7100> FCALL strict 0(error) ERR Script attempted to access keys that do not hash to the same slot script: strict, on @user_function:3.The simplest habit: pass every key in KEYS, give them a common hash tag, and
start the script with #!lua. A script or a function library is stored on the
node you load it on, not on the cluster: see
What cannot be clustered.
Blocking commands
Section titled “Blocking commands”A blocking command such as BLPOP with several keys needs
them in one slot, like any multi-key command. With a common tag it is routed to
the node that serves them:
127.0.0.1:7100> BLPOP q1 q2 0.2(error) CROSSSLOT Keys in request don't hash to the same slot127.0.0.1:7100> BLPOP {q}1 {q}2 0.2-> Redirected to slot [11958] located at 127.0.0.1:7102(nil)Pub/Sub
Section titled “Pub/Sub”PUBLISH is not tied to a slot: it is forwarded to the
subscribers of the channel on every node, so a subscriber can be connected to
any node. The number it returns counts only the subscribers on the node you
sent it to. Sharded Pub/Sub
(SPUBLISH, SSUBSCRIBE)
keeps a channel on the node that serves its slot, and other nodes answer
MOVED.
With a subscriber on port 7101 (SUBSCRIBE news) and one on port 7100
(SSUBSCRIBE {user:1000}news):
127.0.0.1:7100> PUBLISH news hello(integer) 0127.0.0.1:7100> SPUBLISH {user:1000}news hello(integer) 1127.0.0.1:7101> SPUBLISH {user:1000}news hello(error) MOVED 1649 127.0.0.1:7100$ redis-cli --no-raw -p 7101 SUBSCRIBE news1) "subscribe"2) "news"3) (integer) 11) "message"2) "news"3) "hello"$ redis-cli --no-raw -p 7100 SSUBSCRIBE {user:1000}news1) "ssubscribe"2) "{user:1000}news"3) (integer) 11) "smessage"2) "{user:1000}news"3) "hello"Databases
Section titled “Databases”By default a cluster node has one database, as in Redis Cluster, and the commands that move data between databases are refused:
127.0.0.1:7100> SELECT 1(error) ERR SELECT is not allowed in cluster mode127.0.0.1:7100> MOVE user:1000 1(error) ERR MOVE is not allowed in cluster mode127.0.0.1:7100> COPY {user:1000}.a {user:1000}.b DB 1(error) ERR Copying to another database is not allowed in cluster mode127.0.0.1:7100> SWAPDB 0 1(error) ERR SWAPDB is not allowed in cluster mode127.0.0.1:7100> CONFIG GET cluster-databases1) "cluster-databases"2) "1"cluster-databases gives a node 1 to 128 databases. Set it on every node when
you start them; it cannot be changed with
CONFIG SET. On the command line, a value outside 1 to 128 stops the start. In
the configuration file it is only logged and the node runs with 1 database, so
check the log. With a value above 1, SELECT, MOVE and COPY ... DB work (the key keeps its slot in every
database), an index out of range is refused, and SWAPDB
is still refused. This cluster was started with --cluster-databases 4:
127.0.0.1:7110> CONFIG GET cluster-databases1) "cluster-databases"2) "4"127.0.0.1:7110> SELECT 1OK127.0.0.1:7110[1]> SET {u}k v1-> Redirected to slot [11826] located at 127.0.0.1:7112OK127.0.0.1:7112[1]> SELECT 0OK127.0.0.1:7112> SET {u}k v0OK127.0.0.1:7112> MOVE {u}k 1(integer) 0127.0.0.1:7112> SELECT 1OK127.0.0.1:7112[1]> COPY {u}k {u}k2 DB 3(integer) 1127.0.0.1:7112[1]> SELECT 3OK127.0.0.1:7112[3]> GET {u}k2"v1"127.0.0.1:7112[3]> SWAPDB 0 1(error) ERR SWAPDB is not allowed in cluster mode127.0.0.1:7112[3]> SELECT 9(error) ERR DB index is out of rangeMOVE answered 0 because database 1 already held {u}k. The AI engines are
not in a database: an entry is found by its name whichever database you
selected (see AI engines in a cluster).
Counting, listing and scanning
Section titled “Counting, listing and scanning”These commands answer for the node you send them to, as in Redis Cluster:
DBSIZE, KEYS,
SCAN, RANDOMKEY,
FLUSHDB, FLUSHALL and
INFO keyspace. Thirty keys written through the cluster are spread over the
primaries:
$ for p in 7100 7101 7102; do echo "port $p: $(redis-cli -p $p DBSIZE)"; doneport 7100: 18port 7101: 9port 7102: 9$ redis-cli -p 7100127.0.0.1:7100> DBSIZE(integer) 18127.0.0.1:7100> KEYS key:1*1) "key:19"2) "key:15"3) "key:11"4) "key:18"5) "key:14"127.0.0.1:7100> INFO keyspace# Keyspacedb0:keys=18,expires=0,avg_ttl=0,subexpiry=0FLUSHDB and FLUSHALL are just as local. On a cluster that held 30 keys,
emptying the node on port 7101 left the others alone:
127.0.0.1:7101> FLUSHDBOK127.0.0.1:7101> DBSIZE(integer) 0$ for p in 7100 7101 7102; do echo "port $p: $(redis-cli -p $p DBSIZE)"; doneport 7100: 13port 7101: 0port 7102: 8To go through every key of the cluster, use CLUSTERSCAN.
It takes a cursor like SCAN, and the command goes from node to node as the scan
advances. Pass back the cursor
you were given exactly as it is, and stop when it is 0. With
redis-cli -c following the redirects, this script prints every key once:
#!/bin/bashcursor=0while :; do reply=$(redis-cli -c -p 7100 CLUSTERSCAN "$cursor" COUNT 100) cursor=$(printf '%s\n' "$reply" | head -n 1) printf '%s\n' "$reply" | tail -n +2 [ "$cursor" = 0 ] && breakdone$ bash clusterscan.sh | sort -V | head -3citykey:1key:2$ bash clusterscan.sh | wc -l36$ bash clusterscan.sh | sort | uniq -d | wc -l0Every key is listed once.
Reading from replicas
Section titled “Reading from replicas”A replica answers MOVED for every key, even for the ones its primary serves,
until the connection sends READONLY. Then it serves
reads for the slots of its own primary, and still refuses writes and every
other slot. READWRITE turns it off. Replication is asynchronous, so a
replica can be a moment behind: WAIT on the primary
tells you when a write has reached it. Port 7105 is the replica of the
primary on 7102 here:
127.0.0.1:7102> SET city LisbonOK127.0.0.1:7102> WAIT 1 1000(integer) 1127.0.0.1:7102> ROLE1) "master"2) (integer) 703) 1) 1) "127.0.0.1" 2) "7105" 3) "70"127.0.0.1:7105> GET city(error) MOVED 11479 127.0.0.1:7102127.0.0.1:7105> READONLYOK127.0.0.1:7105> GET city"Lisbon"127.0.0.1:7105> SET city Porto(error) MOVED 11479 127.0.0.1:7102127.0.0.1:7105> GET user:1000(error) MOVED 1649 127.0.0.1:7100127.0.0.1:7105> READWRITEOK127.0.0.1:7105> GET city(error) MOVED 11479 127.0.0.1:7102Cluster-aware client libraries have an option that sends the READONLY for
you and spreads reads over replicas: see Cluster clients.
What is clustered and what cannot be
Section titled “What is clustered and what cannot be”Routed by key or name
Section titled “Routed by key or name”These commands go to the node that serves the slot of their key. A key moves
with its slot when you reshard. Sent to a node that does
not serve the slot, a command of each family below is answered with
MOVED to the slot of its key, or served by the node that owns it:
HSET h:1 f v slot 16102 -> MOVED 16102 127.0.0.1:7102LPUSH l:1 a slot 8742 -> MOVED 8742 127.0.0.1:7101SADD s:1 a slot 3444 -> 1ZADD z:1 1 a slot 5093 -> 1XADD x:1 * f v slot 15749 -> MOVED 15749 127.0.0.1:7102GEOADD g:1 13.4 52.5 berlin slot 4823 -> 1SETBIT b:1 1 1 slot 14631 -> MOVED 14631 127.0.0.1:7102PFADD p:1 a slot 5156 -> 1BF.ADD bf:1 a slot 4445 -> 1CMS.INITBYDIM cms:1 100 5 slot 6702 -> MOVED 6702 127.0.0.1:7101TOPK.RESERVE tk:1 3 slot 12082 -> MOVED 12082 127.0.0.1:7102GCRA.LIMIT gc:1 60 5 slot 1768 -> 0(Each command is sent to port 7100, which serves slots 0 to 5460.)
| What | Routed by |
|---|---|
| Strings, hashes (with field expiry), lists, sets, sorted sets, streams, geospatial, bitmaps, HyperLogLog, Bloom filters, Count-Min Sketch, Top-K | the key |
GCRA.* |
the key |
The AI engines, for example PCACHE.*, SCACHE.*, VSIM.*, RAG.*, AMEM.*, LLMBUDGET.*, RATELIMIT.*, PROMPT.* and TS.* |
the name of the entry (cache namespace, index, pipeline, agent or session id, …) |
The second part of the table is covered in AI engines in a cluster, which shows the routing of each family.
What cannot be clustered
Section titled “What cannot be clustered”Some state belongs to a node, and some commands answer from one node only. The cluster does not hide any of it. The transcripts follow the table.
| Per node | What you see | What to do |
|---|---|---|
DBSIZE, KEYS, SCAN, RANDOMKEY, FLUSHDB, FLUSHALL, INFO keyspace |
Only the node you ask. | Ask every primary, or use CLUSTERSCAN for keys. |
The LIST commands of the AI engines, TS.LIST, VSIM.INFO without an index name |
Only the entries on that node. | Ask every primary and merge the answers. |
Scripts (the SCRIPT LOAD cache) |
Cached on the node that loaded them, not on its replicas. | Load them on every primary that runs them, and send them again after a failover, when EVALSHA answers NOSCRIPT. |
Function libraries (FUNCTION LOAD) |
On the node that loaded them and on its replicas, not on the other primaries. | Load them on every primary. |
ACL users (ACL SETUSER) |
Created on that node only, not even on its replicas. | Run the ACL commands on every node, replicas included. |
CONFIG, INFO |
The node you ask. | Send them to every node. |
The TAG.* index |
Stays on the node where it was written. | See below. |
Scripts, functions, ACL users and CONFIG in practice:
127.0.0.1:7100> SCRIPT LOAD "return redis.call('GET', KEYS[1])""d3c21d0c2b9ca22f82737626a27bcaf5d288f99f"127.0.0.1:7102> EVALSHA d3c21d0c2b9ca22f82737626a27bcaf5d288f99f 1 foo(error) NOSCRIPT No matching script. Please use EVAL.127.0.0.1:7102> FCALL get_one 1 foo(error) ERR Function not found127.0.0.1:7100> ACL SETUSER reporter on nopass ~report:* +getOK127.0.0.1:7101> ACL GETUSER reporter(nil)127.0.0.1:7100> ROLE1) "master"2) (integer) 11853) 1) 1) "127.0.0.1" 2) "7103" 3) "1185"127.0.0.1:7103> ACL GETUSER reporter(nil)127.0.0.1:7100> CONFIG SET maxmemory-policy allkeys-lruOK127.0.0.1:7101> CONFIG GET maxmemory-policy1) "maxmemory-policy"2) "noeviction"TS.LIST and VSIM.INFO are examples of listings that answer per node. On a
cluster with several time series and indexes, each primary lists its own:
$ for p in 7003 7004 7005; do echo "port $p: $(redis-cli -p $p VSIM.INFO | tr "\n" " ")"; doneport 7003: productsport 7004: {docs}:indexport 7005:$ for p in 7003 7004 7005; do echo "port $p: $(redis-cli -p $p TS.LIST | tr "\n" " ")"; doneport 7003: sensor-2:rawport 7004: sensor-2:1m sensor-1port 7005: {sensor-3}:raw {sensor-3}:1mThe key tags (TAG.SET, TAG.GET, TAG.MEMBERS) are routed by key, but they
do not move with a reshard: after CLUSTER MIGRATESLOTS moves the slot of
{tag}k, the key is on the new node and its tags are gone. Keep labels that
must follow a key in an ordinary set, which is a key and moves with its slot:
127.0.0.1:7100> SET {tag}k 1-> Redirected to slot [8338] located at 127.0.0.1:7101OK127.0.0.1:7101> TAG.SET {tag}k red(integer) 1127.0.0.1:7101> TAG.GET {tag}k1) "red"127.0.0.1:7101> CLUSTER KEYSLOT {tag}k(integer) 8338$ redis-cli -p 7101 CLUSTER MIGRATESLOTS SLOTSRANGE 8338 8338 NODE $ID0OK127.0.0.1:7100> GET {tag}k"1"127.0.0.1:7100> TAG.GET {tag}k(empty array)127.0.0.1:7101> TAG.MEMBERS red(empty array)127.0.0.1:7100> SET {tag}k 1-> Redirected to slot [8338] located at 127.0.0.1:7101OK127.0.0.1:7101> TAG.SET {tag}k red(integer) 1127.0.0.1:7101> DEL {tag}k(integer) 1127.0.0.1:7101> TAG.MEMBERS red1) "{tag}k"($ID0 is the node ID of the node on port 7100, from
CLUSTER MYID.)
Next steps
Section titled “Next steps”- Cluster clients: connect an application, with retries through failover and reshard.
- AI engines in a cluster: where each entry lives and how to keep related ones together.
- Scaling a cluster: reshard, add and remove nodes.
Was this page helpful?
Thanks for the feedback.