Cluster clients
A cluster-aware client library learns which node serves which slot by asking the cluster, sends each command to the right node, and follows the redirects when the layout changes. HotKV speaks the Redis Cluster protocol, so the cluster libraries you already use work against it, and the HotKV SDKs add typed commands for the AI engines on top.
The snippets on this page assume a cluster of three primaries and
three replicas on ports 7000 to 7005, protected by a password. Start the
nodes as in the quick start, adding
--requirepass cluster-password --masterauth cluster-password to each node
and the password to the redis-cli --cluster create command. Replace the
addresses and the password with yours.
How a cluster client works
Section titled “How a cluster client works”- Seed nodes. You give the client a few node addresses. It asks one of them
for the slot map (
CLUSTER SLOTS), then connects to the nodes it learns about. Give two or three seeds in case one is down at start. Seeds are used only to find the cluster: the client talks to the addresses the cluster announces. - The announced address must be reachable from the client.
MOVEDreplies andCLUSTER SLOTScarry the address each node announces, and the client connects to exactly that. A cluster whose nodes announce a container or private address works from inside that network only. See Deploy on VMs and cloud for the announce settings, and the Compose and Kubernetes pages for containers. - Routing by key. The client hashes the key to a slot, so a command with
several keys needs them in one slot: use a
hash tag such as
{user:1000}.name. A multi-key command across slots fails withCROSSSLOT. - Password and TLS are connection settings applied to every node. TLS options are covered in Cluster TLS.
- Client-side caching is per node. A client that keeps a local cache turns
CLIENT TRACKINGon for each node connection it reads from, and is told of the changes that node makes to the keys it read. A node that gives up a slot tells such clients of each key as it removes them, so a cache can drop what it held from that node. See Client-side caching.
What the redirects and errors mean to a client
Section titled “What the redirects and errors mean to a client”| Reply | Meaning | What a good client does |
|---|---|---|
MOVED slot address | The slot is served by another node. | Refresh the slot map and resend to that address. |
ASK slot address | The slot is being moved and this key is already on the target. | Send ASKING and the command to that address, once. Do not update the slot map. |
CLUSTERDOWN | The slot is not served, or the cluster is down (for example, during a failover that has not finished). | Wait briefly, refresh the map and retry. |
TRYAGAIN | A multi-key command found its keys split across a slot that is moving. An AI-engine entry of a slot that this node is handing off gets it too. | Retry after a short pause. |
LOADING | The node is loading or syncing its data. | Retry, or use another node. |
CROSSSLOT | The keys of a command are in different slots. | Not retryable: fix the keys (hash tags). |
Connect and use
Section titled “Connect and use”Each snippet connects with two seed nodes and the password, writes and reads a key, runs a multi-key command on keys that share a hash tag, and reads from replicas with a second client. The output of each one is shown under it.
redis-py (works with 8.1.0 and later): pip install redis, in a virtual environment (python3 -m venv venv && . venv/bin/activate) on systems whose Python is managed by the operating system.
from redis.cluster import RedisCluster, ClusterNode
rc = RedisCluster( startup_nodes=[ClusterNode("127.0.0.1", 7000), ClusterNode("127.0.0.1", 7001)], password="cluster-password", decode_responses=True,)
rc.set("user:1000", "ada")print(rc.get("user:1000"))
# Keys with a common hash tag live in one slot, so a multi-key command works.rc.mset({"{user:1000}.name": "Ada", "{user:1000}.city": "London"})print(rc.mget("{user:1000}.name", "{user:1000}.city"))
# Keys in different slots need one command each.rc.mset_nonatomic({"user:1": "a", "user:2": "b", "user:3": "c"})print(rc.mget_nonatomic("user:1", "user:2", "user:3"))
# Read from replicas: a second client that sends READONLY to replica nodes.rr = RedisCluster( startup_nodes=[ClusterNode("127.0.0.1", 7000)], password="cluster-password", decode_responses=True, read_from_replicas=True,)print(rr.get("user:1000"))rc.close()rr.close()ada['Ada', 'London']['a', 'b', 'c']adaread_from_replicas=True sends reads to the replicas of the slot’s primary
(and the library sends READONLY to them). Without it every command goes to
the primary.
node-redis (works with 6.3.0 and later): npm install redis.
import { createCluster } from 'redis';
const options = { rootNodes: [{ url: 'redis://127.0.0.1:7000' }, { url: 'redis://127.0.0.1:7001' }], defaults: { password: 'cluster-password' },};
const cluster = createCluster(options);cluster.on('error', (err) => console.error('cluster error:', err.message));await cluster.connect();
await cluster.set('user:1000', 'ada');console.log(await cluster.get('user:1000'));
// Keys with a common hash tag live in one slot, so a multi-key command works.await cluster.mSet({ '{user:1000}.name': 'Ada', '{user:1000}.city': 'London' });console.log(await cluster.mGet(['{user:1000}.name', '{user:1000}.city']));
// Read from replicas: a second client with useReplicas.const replicas = createCluster({ ...options, useReplicas: true });replicas.on('error', (err) => console.error('cluster error:', err.message));await replicas.connect();console.log(await replicas.get('user:1000'));
await replicas.close();await cluster.close();ada[ 'Ada', 'London' ]adaioredis (works with 6.0.0 and later): npm install ioredis.
import { Cluster } from 'ioredis';
const nodes = [{ host: '127.0.0.1', port: 7000 }, { host: '127.0.0.1', port: 7001 }];const redisOptions = { password: 'cluster-password' };
const cluster = new Cluster(nodes, { redisOptions });
await cluster.set('user:1000', 'ada');console.log(await cluster.get('user:1000'));
// Keys with a common hash tag live in one slot, so a multi-key command works.await cluster.mset('{user:1000}.name', 'Ada', '{user:1000}.city', 'London');console.log(await cluster.mget('{user:1000}.name', '{user:1000}.city'));
// Read from replicas: a second client with scaleReads.const replicas = new Cluster(nodes, { redisOptions, scaleReads: 'slave' });console.log(await replicas.get('user:1000'));
replicas.disconnect();cluster.disconnect();ada[ 'Ada', 'London' ]adaIn node-redis the client options for the nodes (password, TLS) go in
defaults, because the nodes it discovers do not inherit the options of the
root nodes. useReplicas: true in node-redis and scaleReads: 'slave' in
ioredis send reads to replicas.
go-redis (works with v9.22.0 and later): go get github.com/redis/go-redis/v9.
package main
import ( "context" "fmt"
"github.com/redis/go-redis/v9")
func main() { ctx := context.Background() options := &redis.ClusterOptions{ Addrs: []string{"127.0.0.1:7000", "127.0.0.1:7001"}, Password: "cluster-password", }
rdb := redis.NewClusterClient(options) defer rdb.Close()
rdb.Set(ctx, "user:1000", "ada", 0) fmt.Println(rdb.Get(ctx, "user:1000").Val())
// Keys with a common hash tag live in one slot, so a multi-key command works. rdb.MSet(ctx, "{user:1000}.name", "Ada", "{user:1000}.city", "London") fmt.Println(rdb.MGet(ctx, "{user:1000}.name", "{user:1000}.city").Val())
// Read from replicas: a second client with ReadOnly (reads go to the primary or its replicas). replicaOptions := *options replicaOptions.ReadOnly = true replicaOptions.RouteRandomly = true rr := redis.NewClusterClient(&replicaOptions) defer rr.Close() fmt.Println(rr.Get(ctx, "user:1000").Val())}ada[Ada London]adaReadOnly: true lets the client read from replicas, and RouteRandomly: true
spreads those reads over the primary and its replicas.
Jedis (works with 5.2.0 and later): Maven redis.clients:jedis:5.2.0.
import java.util.List;import java.util.Set;import redis.clients.jedis.DefaultJedisClientConfig;import redis.clients.jedis.HostAndPort;import redis.clients.jedis.JedisCluster;
public class JedisBasic { public static void main(String[] args) { Set<HostAndPort> seeds = Set.of(new HostAndPort("127.0.0.1", 7000), new HostAndPort("127.0.0.1", 7001)); DefaultJedisClientConfig config = DefaultJedisClientConfig.builder().password("cluster-password").build();
try (JedisCluster jc = new JedisCluster(seeds, config)) { jc.set("user:1000", "ada"); System.out.println(jc.get("user:1000"));
// Keys with a common hash tag live in one slot, so a multi-key command works. jc.mset("{user:1000}.name", "Ada", "{user:1000}.city", "London"); List<String> values = jc.mget("{user:1000}.name", "{user:1000}.city"); System.out.println(values); } }}ada[Ada, London]Lettuce (works with 6.5.0 and later): Maven io.lettuce:lettuce-core:6.5.0.RELEASE.
import io.lettuce.core.ReadFrom;import io.lettuce.core.RedisURI;import io.lettuce.core.cluster.RedisClusterClient;import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;import io.lettuce.core.cluster.api.sync.RedisAdvancedClusterCommands;import java.util.List;
public class LettuceBasic { public static void main(String[] args) { char[] password = "cluster-password".toCharArray(); RedisClusterClient client = RedisClusterClient.create(List.of( RedisURI.Builder.redis("127.0.0.1", 7000).withPassword(password).build(), RedisURI.Builder.redis("127.0.0.1", 7001).withPassword(password).build()));
try (StatefulRedisClusterConnection<String, String> connection = client.connect()) { RedisAdvancedClusterCommands<String, String> commands = connection.sync(); commands.set("user:1000", "ada"); System.out.println(commands.get("user:1000"));
// Keys with a common hash tag live in one slot, so a multi-key command works. commands.mset(java.util.Map.of("{user:1000}.name", "Ada", "{user:1000}.city", "London")); System.out.println(commands.mget("{user:1000}.name", "{user:1000}.city"));
// Read from replicas. connection.setReadFrom(ReadFrom.REPLICA_PREFERRED); System.out.println(commands.get("user:1000")); } finally { client.shutdown(); } }}ada[KeyValue[{user:1000}.name, Ada], KeyValue[{user:1000}.city, London]]adaLettuce reads from replicas with connection.setReadFrom(...). The Jedis
snippet has no replica reads: it uses the primaries only.
Survive a failover
Section titled “Survive a failover”A client in a cluster should ride out a failover: the slot moves to another
node, the old connection closes, and for a moment commands for those slots get
CLUSTERDOWN or a refused connection. A cluster-aware client refreshes its slot
map and retries those commands, so a coordinated failover (CLUSTER FAILOVER,
docker compose stop with --shutdown-on-sigterm failover, a rolling update)
pauses writes for a moment and hands the slot over without losing a write.
A crash is different: the slot has no primary until the other nodes agree that it is gone, which takes the node timeout (15 seconds by default) plus the election, and a client that gives up sooner reports errors in that time. See Replicas and failover for the timeline. How long a client library retries before it raises an error is a setting of that library. Set the retry count and timeout of your library so that they cover the node timeout plus the election.
HotKV SDK cluster clients
Section titled “HotKV SDK cluster clients”The Python and Node.js SDKs have a cluster client that routes every command,
including the typed commands of the AI engines, by the key or name it carries,
follows MOVED and ASK, and rides out a failover, a reshard in progress,
CLUSTERDOWN, TRYAGAIN and LOADING by refreshing the slot map and trying
again for up to 10 seconds (retry_timeout in Python, retryTimeoutMs in
Node.js) before it raises ClusterError. A command that may have reached a
node whose connection then broke is sent again, so a command that is not
idempotent (incr, rpush, a budget consume) can be applied twice; pass
retry_writes=False (retryWrites: false) to get the error instead. Install
them as shown on Connecting and SDKs.
ClusterHotKV, in the hotkv.cluster module:
from hotkv.cluster import ClusterHotKV
cluster = ClusterHotKV( [("127.0.0.1", 7000), ("127.0.0.1", 7001)], password="cluster-password", decode_responses=True,)
cluster.set("user:1000", "ada")print(cluster.get("user:1000"))
# Keys with a common hash tag live in one slot, so a multi-key command works.print(cluster.execute("MSET", "{user:1000}.name", "Ada", "{user:1000}.city", "London"))print(cluster.execute("MGET", "{user:1000}.name", "{user:1000}.city"))
# Commands that list one node's data are sent to every primary and merged.print(cluster.dbsize())print(sorted(cluster.keys("{user:1000}*")))
# A command for one node's own state goes where you say.print(cluster.execute_on_primaries("CONFIG", "GET", "maxmemory-policy"))
# Read from replicas: replica connections send READONLY.replicas = ClusterHotKV( [("127.0.0.1", 7000)], password="cluster-password", decode_responses=True, read_from_replicas=True,)print(replicas.get("user:1000"))replicas.close()cluster.close()adaTrue['Ada', 'London']103['{user:1000}.city', '{user:1000}.name']{('127.0.0.1', 7000): {'maxmemory-policy': 'noeviction'}, ('127.0.0.1', 7001): {'maxmemory-policy': 'noeviction'}, ('127.0.0.1', 7002): {'maxmemory-policy': 'noeviction'}}adaThe asyncio client has the same methods as coroutines:
import asynciofrom hotkv.asyncio import ClusterHotKV
async def main(): cluster = ClusterHotKV( [("127.0.0.1", 7000), ("127.0.0.1", 7001)], password="cluster-password", decode_responses=True, ) await cluster.set("user:1000", "ada") print(await cluster.get("user:1000")) print(await cluster.execute("MGET", "{user:1000}.name", "{user:1000}.city")) await cluster.close()
asyncio.run(main())ada['Ada', 'London']HotKVCluster, exported by the hotkv package:
import { HotKVCluster } from '@hotkv/client';
const cluster = new HotKVCluster(['127.0.0.1:7000', '127.0.0.1:7001'], { password: 'cluster-password',});
await cluster.set('user:1000', 'ada');console.log(await cluster.get('user:1000'));
// Keys with a common hash tag live in one slot, so a multi-key command works.console.log(await cluster.execute(['MSET', '{user:1000}.name', 'Ada', '{user:1000}.city', 'London']));const values = await cluster.execute(['MGET', '{user:1000}.name', '{user:1000}.city']);console.log(values.map((v) => v.toString()));
// Commands that list one node's data are sent to every primary and merged.console.log(await cluster.dbSize());
// A command for one node's own state goes to every primary when you ask.const perNode = await cluster.executeOnPrimaries(['CONFIG', 'GET', 'maxmemory-policy']);console.log([...perNode.keys()]);
// Read from replicas: replica connections send READONLY.const replicas = new HotKVCluster(['127.0.0.1:7000'], { password: 'cluster-password', readFromReplicas: true,});console.log(await replicas.get('user:1000'));
await replicas.close();await cluster.close();adaOK[ 'Ada', 'London' ]103[ '127.0.0.1:7000', '127.0.0.1:7001', '127.0.0.1:7002' ]adaCommands that answer for one node
Section titled “Commands that answer for one node”Some commands report on one node’s own data: the LIST commands of the AI
engines, KEYS, DBSIZE, FLUSHALL, TAG.MEMBERS, and the script and
function commands (see
What cannot be clustered).
The HotKV cluster clients send these to every primary and merge the answers,
so they cover the whole cluster: dbsize() above counted the keys of all
three primaries (the number is however many keys your cluster holds), and pcache_list() lists the prompt-cache namespaces of the
whole cluster:
from hotkv.cluster import ClusterHotKV
cluster = ClusterHotKV([("127.0.0.1", 7000), ("127.0.0.1", 7001)], decode_responses=True)
for ns in ("notes", "faqs", "plans", "agents"): cluster.pcache_create(ns) cluster.pcache_set(ns, "question", "answer")print(cluster.pcache_get("faqs", "question"))
# pcache_list() asks every primary and merges, so it lists the whole cluster.print(sorted(cluster.pcache_list()))cluster.close()answer['agents', 'faqs', 'notes', 'plans'](The AI engines need a license on every node: see AI engines in a cluster.)
State that belongs to one node, such as an ACL user, is not merged: the client
sends ACL, CONFIG, INFO and similar commands to one primary, and you send
them to a node you choose with execute_on_node (Python) or executeOnNode
(Node.js), or to every primary with execute_on_primaries or
executeOnPrimaries, which return the answer of each node, as the
CONFIG GET calls in the snippets above did. An ACL user created on one
node does not exist on the others, or on its replicas.
Other HotKV SDKs
Section titled “Other HotKV SDKs”Each of these SDKs has a cluster client that builds its slot map from the
cluster and follows MOVED and ASK. Install them as shown on
Connecting and SDKs.
| Language | Cluster class | Replica reads |
|---|---|---|
| Java | com.hotkv.HotKVCluster |
commandReadOnly |
| C# | HotKV.Cluster.HotKVCluster |
ReadFromReplicas = true |
| C++ | hotkv::ClusterClient |
set_read_from_replicas(true) |
| PHP | HotKV\Cluster\ClusterClient |
the readFromReplicas option |
| Ruby | HotKV::Cluster |
- |
| Rust | hotkv::ClusterClient |
ClusterOptions::read_from_replicas with execute_readonly |
| Elixir | HotKV.Cluster |
- |
| Perl | HotKV::Cluster |
the read_from_replicas option |
The Go and C SDKs have a function that computes the slot of a key
(KeySlot, hotkv_key_slot) and recognise redirect errors. In Go, use go-redis’s ClusterClient as shown above.
Next steps
Section titled “Next steps”- Keys, slots and commands: hash tags and what answers per node.
- Replicas and failover: what a client sees during a failover.
- Cluster TLS: the connection options for TLS.
Was this page helpful?
Thanks for the feedback.