# AI engines in a cluster

> Where the cache, vector, memory, budget and time series engines keep their entries in a cluster, what survives a failover, and how to keep related entries together.

URL: https://docs.hotkv.com/clustering/ai-engines/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

HotKV's AI engines (the prompt, semantic, tool, negative, guard and rerank
caches, vector search, RAG pipelines, agent memory, LLM budgets, statistics
and rate limits, the prompt registry, embedding de-duplication, the feature
store and time series) are clustered like keys: every entry has a name, the
name decides the hash slot, and the entry lives on the node that serves that
slot. You use the same commands as on a single server, and a cluster-aware
client sends each one to the right node. This page shows where each kind of
entry goes, what a replica and a failover keep, how to put entries that belong
together on one node, which state is shared by all nodes and which is not,
and what a reshard does with entries.

The examples use six nodes (three primaries, three replicas) on ports 7000
to 7005. Cluster mode needs no license, but the engines do: give every node
the license file with the `license-path` setting, as described in
[AI features](https://docs.hotkv.com/ai-features/index.md#licensing-and-plans). The license belongs to the
node, so `LICENSE.CHECK` answers for the node you ask:

```text
$ redis-cli -p 7000 LICENSE.CHECK promptcache; redis-cli -p 7003 LICENSE.CHECK promptcache
1
1
```

## Where an entry lives

An entry is found by its name: the cache namespace, the index name, the
pipeline name, the agent or session id, the budget, bucket or limiter name,
the template name, the series name. `CLUSTER KEYSLOT` of that name is the slot,
and a node that does not serve the slot answers `MOVED` for the command, just
as it does for a key.

| Commands | Routed by |
| --- | --- |
| `PCACHE.*`, `TOOL.CACHE.*`, `NCACHE.*`, `GUARD.*`, `RERANK.*` | the namespace |
| `SCACHE.*` | the namespace |
| `VSIM.*` (indexes), except `VSIM.SAVE`, `VSIM.LOAD`, `VSIM.DROP` and `VSIM.LIST` | the index name |
| `RAG.*` | the pipeline name |
| `AMEM.CONV.*`, `AMEM.WORK.*` | the session id |
| `AMEM.SEM.*`, `AMEM.EPIS.*`, `AMEM.SEARCH`, `AMEM.SUMMARY.*` | the agent id |
| `LLMBUDGET.*`, `RATELIMIT.*` | the budget or limiter name |
| `LLMSTATS.*` (buckets and records) | the bucket name |
| `PROMPT.*`, `EMBED.DEDUP.*`, `FEATURE.*`, `TS.*` | the template, namespace, feature group or series name |

On a node that does not serve the slot of a name, the command is refused with
`MOVED` and the slot you get from `CLUSTER KEYSLOT`. `redis-cli -c` follows
the redirect:

```text
127.0.0.1:7000> CLUSTER KEYSLOT faq
(integer) 16230
127.0.0.1:7000> SCACHE.CREATE faq 3
(error) MOVED 16230 127.0.0.1:7002
127.0.0.1:7000> CLUSTER KEYSLOT products
(integer) 10285
127.0.0.1:7000> VSIM.CREATEINDEX products 3 METRIC COSINE
(error) MOVED 10285 127.0.0.1:7001
127.0.0.1:7000> CLUSTER KEYSLOT api-limit
(integer) 11870
127.0.0.1:7000> RATELIMIT.CREATE api-limit 100 60000
(error) MOVED 11870 127.0.0.1:7002
```

```text
$ redis-cli -c -p 7000
127.0.0.1:7000> PCACHE.CREATE prompts
OK
127.0.0.1:7000> SCACHE.CREATE faq 3
-> Redirected to slot [16230] located at 127.0.0.1:7002
OK
127.0.0.1:7002> VSIM.CREATEINDEX products 3 METRIC COSINE
-> Redirected to slot [10285] located at 127.0.0.1:7001
OK
127.0.0.1:7001> RATELIMIT.CREATE api-limit 100 60000
-> Redirected to slot [11870] located at 127.0.0.1:7002
OK
127.0.0.1:7002> LLMBUDGET.SET team-budget 100000 5.0 3600
-> Redirected to slot [521] located at 127.0.0.1:7000
OK
127.0.0.1:7000> RAG.CREATE support-docs 3
-> Redirected to slot [10518] located at 127.0.0.1:7001
OK
127.0.0.1:7001> AMEM.CONV.CREATE sess-1
-> Redirected to slot [16334] located at 127.0.0.1:7002
OK
127.0.0.1:7002> AMEM.SEM.CREATE agent-7 3
OK
127.0.0.1:7002> TS.CREATE sensor-1
OK
127.0.0.1:7002> PROMPT.PUT greeting "Hello {{name}}"
(integer) 1
```

The remaining families route the same way: each command is answered by the
primary that serves the slot of the name.

| Command | Name | Slot |
| --- | --- | --- |
| `TOOL.CACHE.CREATE` | `tools` | 11861 |
| `NCACHE.CREATE` | `misses` | 545 |
| `GUARD.CREATE` | `safety` | 7426 |
| `RERANK.CREATE` | `ranks` | 12077 |
| `LLMSTATS.RECORD` | `api-bucket` | 5653 |
| `AMEM.WORK.SET` | `sess-9` | 16070 |
| `AMEM.EPIS.LOG`, `AMEM.SUMMARY.SET` | `agent-9` | 3625 |
| `AMEM.SEARCH` | `agent-7` | 12263 |

A cluster-aware client does this routing for you. The Python and Node.js
HotKV client libraries route every command in the table by its name: see
[Cluster clients](https://docs.hotkv.com/clustering/clients/index.md).

An entry is never split between nodes. A namespace, an index, a limiter, a
budget, a time series or an agent's memory is held by exactly one node, so
its size and its request rate are bounded by that node. To spread the load,
use several names, for example one cache namespace per tenant. An entry larger
than 512 MB cannot be moved by a reshard.

## Keep related entries together

Some entries belong together but have different names, and so land on
different nodes: an agent's conversation sessions (`AMEM.CONV`, by session id)
and its long-term memory (`AMEM.SEM`, by agent id), or a cache namespace and
the vector index you keep beside it. Give them a common
[hash tag](https://docs.hotkv.com/clustering/keys-and-commands/index.md#hash-tags) and they share a slot,
and so a node, and move together:

```text
$ redis-cli -c -p 7000
127.0.0.1:7000> CLUSTER KEYSLOT sess-2
(integer) 4013
127.0.0.1:7000> CLUSTER KEYSLOT agent-8
(integer) 7688
127.0.0.1:7000> CLUSTER KEYSLOT {agent-8}:sess-2
(integer) 7688
127.0.0.1:7000> CLUSTER KEYSLOT {agent-8}
(integer) 7688
127.0.0.1:7000> AMEM.CONV.CREATE {agent-8}:sess-2
-> Redirected to slot [7688] located at 127.0.0.1:7001
OK
127.0.0.1:7001> AMEM.SEM.CREATE {agent-8} 3
OK
127.0.0.1:7001> CLUSTER KEYSLOT {docs}:cache
(integer) 13556
127.0.0.1:7001> CLUSTER KEYSLOT {docs}:index
(integer) 13556
127.0.0.1:7001> SCACHE.CREATE {docs}:cache 3
-> Redirected to slot [13556] located at 127.0.0.1:7002
OK
127.0.0.1:7002> VSIM.CREATEINDEX {docs}:index 3
OK
```

A RAG pipeline needs no tag for its own vectors: they are kept with the
pipeline, under its name.

[`TS.DOWNSAMPLE`](https://docs.hotkv.com/commands/ts-downsample/index.md) is the one AI command that names
two entries, the source series and the destination series, so they must be in
the same slot, like the keys of a multi-key command. Give the pair a common
tag, and the rule and both series stay on one node:

```text
127.0.0.1:7000> TS.CREATE sensor-2:raw
-> Redirected to slot [5485] located at 127.0.0.1:7001
OK
127.0.0.1:7001> TS.CREATE sensor-2:1m
-> Redirected to slot [12175] located at 127.0.0.1:7002
OK
127.0.0.1:7002> TS.DOWNSAMPLE sensor-2:raw sensor-2:1m avg 60000
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:7002> TS.CREATE {sensor-3}:raw
-> Redirected to slot [5109] located at 127.0.0.1:7000
OK
127.0.0.1:7000> TS.CREATE {sensor-3}:1m
OK
127.0.0.1:7000> TS.DOWNSAMPLE {sensor-3}:raw {sensor-3}:1m avg 60000
OK
```

## Tables every node holds

A few tables are not sharded. Every node holds the whole table: the LLM price
table (`LLMSTATS.PRICE`), the tenants (`TENANT.*`, except `TENANT.STATS`), and the IAM roles and
who has them (`IAM.ROLE.*`, `IAM.USER.ASSIGN`, `IAM.USER.UNASSIGN`). You change them on any
primary, and the change reaches the other nodes. A replica refuses the change, as it
refuses any write.

Here the price table and a tenant are written on one primary, and read from
another:

```text
127.0.0.1:7000> LLMSTATS.PRICE SET gpt-4o INPUT 2.5 OUTPUT 10
OK
127.0.0.1:7000> TENANT.CREATE acme "Acme Corp" MAX_KEYS 1000000
OK
127.0.0.1:7002> LLMSTATS.PRICE GET gpt-4o
 1) "model"
 2) "gpt-4o"
 3) "source"
 4) "custom"
 5) "priced"
 6) (integer) 1
 7) "retired"
 8) (integer) 0
 9) "input_usd_per_mtok"
10) "2.500000"
11) "output_usd_per_mtok"
12) "10.000000"
13) "cache_read_usd_per_mtok"
14) "0.250000"
15) "cache_write_5m_usd_per_mtok"
16) "3.125000"
17) "cache_write_1h_usd_per_mtok"
18) "5.000000"
127.0.0.1:7002> TENANT.LIST
1) "acme"
```

## State that belongs to one node

Some state is per node and the cluster does not copy it:

- **The license.** Each node has its own, as shown above.
- **Encryption at rest, the audit log, `OBSERVE` and `ALERTS`.** Each node
  keeps its own, and they are not copied to the other nodes.
- **Listings.** The `LIST` commands of the engines (`PCACHE.LIST` and the
  others), `TS.LIST` and `VSIM.INFO` without an index name answer for the
  node you ask. To list the whole cluster, ask every primary and merge.
- **Sessions, hot keys and usage counters.** `IAM.SESSION.*` (a login belongs
  to the node it is on), `HOTKEYS`, `TENANT.STATS`, and `INFO`, `CONFIG`,
  `CLIENT`, `SLOWLOG` and `LATENCY` answer for the node you ask.
- **Vector snapshots.** `VSIM.SAVE`, `VSIM.LOAD`, `VSIM.DROP` and `VSIM.LIST` are
  a manual store of named snapshots on the node you send them to. The live
  indexes are the ones that shard.
- **`REPLICATION.*`.** These commands act on one node and are for a standalone
  primary and replica. A cluster promotes with `CLUSTER FAILOVER` and fails over
  on its own.
- **Limits.** The count limits an engine applies to its own commands (the number
  of cache namespaces, limiters, buckets and so on) are checked by each node for
  the entries it creates. An entry a reshard brings in is accepted past the limit,
  so a node can hold more than its limit until some are dropped.
- **Scripts, functions and ACL users**, as on any cluster: see
  [What cannot be clustered](https://docs.hotkv.com/clustering/keys-and-commands/index.md#what-cannot-be-clustered).

```text
$ for p in 7000 7001 7002; do echo "port $p: $(redis-cli -p $p PCACHE.LIST | tr "\n" " ")"; done
port 7000: prompts summaries
port 7001: answers
port 7002: tools
$ for addr in $(redis-cli -p 7000 CLUSTER NODES | awk "/master/ {sub(/@.*/, \"\", \$2); print \$2}"); do redis-cli -h ${addr%:*} -p ${addr#*:} PCACHE.LIST; done | sort
answers
prompts
summaries
tools
$ for p in 7000 7001 7002; do echo "port $p: $(redis-cli -p $p VSIM.INFO | tr "\n" " ")"; done
port 7000:
port 7001: products
port 7002: {docs}:index
$ for p in 7000 7001 7002; do echo "port $p: $(redis-cli -p $p TS.LIST | tr "\n" " ")"; done
port 7000: {sensor-3}:raw {sensor-3}:1m
port 7001: sensor-2:raw
port 7002: sensor-2:1m sensor-1
```

The second command lists the primaries from `CLUSTER NODES` and merges what
each one lists.

The engines are not in databases. An entry is found by its name whatever
database the connection selected, and `FLUSHDB` leaves the entries alone:
see [Databases](https://docs.hotkv.com/clustering/keys-and-commands/index.md#databases).

## Replication and failover

The writes of the engines reach the replicas and the append-only file like
the writes of keys, so a promoted replica has the entries. Write one entry of each of these
engines on its primary (these commands, sent through `redis-cli -c`, add to what
the earlier examples created), fail every primary over with `CLUSTER FAILOVER`
on its replica, one after the other, and read the data back through the new
primaries:

```bash
redis-cli -c -p 7000 <<'EOF'
PCACHE.CREATE answers
PCACHE.CREATE summaries
PCACHE.CREATE tools
PCACHE.SET prompts "capital of France" Paris
SCACHE.SET faq "How do I reset my password?" "Use the account page" 0.1 0.9 0.2
VSIM.SET products sku-1 0.1 0.9 0.2
RATELIMIT.CHECK api-limit user-1
LLMBUDGET.CONSUME team-budget 1200 0.05
TS.ADD sensor-1 1700000000000 21.5
AMEM.SEM.STORE agent-7 fact-1 "Ada likes tea" 0.1 0.2 0.3
AMEM.CONV.APPEND sess-1 user Hello
RAG.INGEST support-docs doc:1 "Reset your password on the login page." VECTORS 0.9,0.1,0.0
EMBED.DEDUP.PUT ns1 h1 model abc
FEATURE.CREATE users "Users" SCHEMA age int64 country string
FEATURE.SET users user:1 age 31 country DE
EOF
```

Then the failover of every primary:

```text
$ for p in 7003 7004 7005; do redis-cli -p $p CLUSTER FAILOVER; until [ "$(redis-cli -p $p ROLE | head -1)" = master ]; do sleep 0.5; done; done
OK
OK
OK
$ redis-cli -p 7000 CLUSTER NODES | awk "{print \$2, \$3, \$9}" | sort
127.0.0.1:7000@17000 myself,slave
127.0.0.1:7001@17001 slave
127.0.0.1:7002@17002 slave
127.0.0.1:7003@17003 master 10923-16383
127.0.0.1:7004@17004 master 0-5460
127.0.0.1:7005@17005 master 5461-10922
```

```text
127.0.0.1:7000> PCACHE.GET prompts "capital of France"
-> Redirected to slot [1197] located at 127.0.0.1:7004
"Paris"
127.0.0.1:7004> SCACHE.GET faq 0.1 0.9 0.2
-> Redirected to slot [16230] located at 127.0.0.1:7003
1) "Use the account page"
2) "1"
127.0.0.1:7003> VSIM.SEARCH products 1 0.1 0.9 0.2
-> Redirected to slot [10285] located at 127.0.0.1:7005
1) "sku-1"
2) "0"
127.0.0.1:7005> RATELIMIT.CHECK api-limit user-1
-> Redirected to slot [11870] located at 127.0.0.1:7003
1) (integer) 1
2) (integer) 98
3) (integer) 0
4) (integer) 100
5) (integer) 1
127.0.0.1:7003> LLMBUDGET.STATUS team-budget
-> Redirected to slot [521] located at 127.0.0.1:7004
 1) "tokens_used"
 2) (integer) 1200
 3) "token_limit"
 4) (integer) 100000
 5) "cost_used_usd"
 6) "0.050000"
 7) "cost_limit_usd"
 8) "5.000000"
 9) "window_seconds"
10) (integer) 3600
11) "window_remaining"
12) (integer) 3591
13) "requests_allowed"
14) (integer) 1
15) "requests_rejected"
16) (integer) 0
17) "unpriced_requests"
18) (integer) 0
127.0.0.1:7004> PROMPT.GET greeting
-> Redirected to slot [12714] located at 127.0.0.1:7003
1) (integer) 1
2) "Hello {{name}}"
3) ""
4) (integer) 1791065805
127.0.0.1:7003> TS.RANGE sensor-1 - +
1) 1) (integer) 1700000000000
   2) "21.500000"
127.0.0.1:7003> AMEM.SEM.RECALL agent-7 1 0.1 0.2 0.3
1) 1) "key"
   2) "fact-1"
   3) "content"
   4) "Ada likes tea"
   5) "similarity"
   6) "0.9999999"
127.0.0.1:7003> AMEM.CONV.GET sess-1
1) 1) "role"
   2) "user"
   3) "content"
   4) "Hello"
127.0.0.1:7003> RAG.SEARCH support-docs "reset password" K 1 VECTOR 0.9,0.1,0.0
-> Redirected to slot [10518] located at 127.0.0.1:7005
1) 1) "doc:1:0"
   2) "doc:1"
   3) "Reset your password on the login page."
   4) "1.000000"
   5) "1.000000"
   6) "0.575364"
127.0.0.1:7005> EMBED.DEDUP.GET ns1 h1
-> Redirected to slot [15122] located at 127.0.0.1:7003
"abc"
127.0.0.1:7003> FEATURE.GET users user:1
1) 1) "age"
   2) (integer) 31
   3) "country"
   4) "DE"
2) (integer) 1791065810600
3) (integer) 4259
```

The prompt cache, semantic cache, vector index, rate limiter, budget, prompt
registry, time series, semantic and conversation memory, RAG pipeline,
de-duplication cache and feature store all answered from the promoted
replica. The rate limiter and the budget kept their counters (98 requests left
after the second check, 1200 tokens used). The price table, written before
the failover, was there too:

```text
127.0.0.1:7004> LLMSTATS.PRICE GET gpt-4o
 1) "model"
 2) "gpt-4o"
 3) "source"
 4) "custom"
 5) "priced"
 6) (integer) 1
 7) "retired"
 8) (integer) 0
 9) "input_usd_per_mtok"
10) "2.500000"
11) "output_usd_per_mtok"
12) "10.000000"
13) "cache_read_usd_per_mtok"
14) "0.250000"
15) "cache_write_5m_usd_per_mtok"
16) "3.125000"
17) "cache_write_1h_usd_per_mtok"
18) "5.000000"
```

### Time to live and clocks

An entry written with a time to live expires by the clock of each node that
holds it. Keep the clocks of all nodes of a cluster in step with NTP: a
replica whose clock runs ahead of its primary's drops entries earlier than the
primary does, and one that runs behind keeps them longer.

## What a reshard does with entries

Engine entries have no key, so `MIGRATE` never sees them. A reshard moves
them along with the slot anyway, whichever tool you use. Each of the four ways
below moves a slot that holds an entry, and the entry is read from the new owner.

**`CLUSTER MIGRATESLOTS`** moves the keys and the entries of the slot in one
atomic step. Run it on the node that serves the slot, with the target's node
ID:

```text
$ redis-cli -p 7000 CLUSTER NODES | awk "/master/ {print \$2, \$9}" | sort
127.0.0.1:7003@17003 10923-16383
127.0.0.1:7004@17004 0-5460
127.0.0.1:7005@17005 5461-10922
$ SRC=$(redis-cli -p 7004 CLUSTER MYID); DST=$(redis-cli -p 7005 CLUSTER MYID)
$ redis-cli -p 7004 CLUSTER MIGRATESLOTS SLOTSRANGE 1197 1197 NODE $DST
OK
$ redis-cli -p 7000 CLUSTER NODES | awk "/master/ {print \$2, \$9, \$10, \$11}" | sort
127.0.0.1:7003@17003 10923-16383
127.0.0.1:7004@17004 0-1196 1198-5460
127.0.0.1:7005@17005 1197 5461-10922
127.0.0.1:7000> PCACHE.GET prompts "capital of France"
-> Redirected to slot [1197] located at 127.0.0.1:7005
"Paris"
```

**`valkey-cli --cluster reshard --cluster-use-atomic-slot-migration`** asks
the nodes for the same atomic migration. The slot it moves first is the lowest
slot of the source; the name `06S` is in slot 0, which is why the example uses
it:

```text
127.0.0.1:7000> CLUSTER KEYSLOT 06S
(integer) 0
127.0.0.1:7000> PCACHE.CREATE 06S
-> Redirected to slot [0] located at 127.0.0.1:7004
OK
127.0.0.1:7004> PCACHE.SET 06S question answer
OK
$ SRC=$(redis-cli -p 7004 CLUSTER MYID); DST=$(redis-cli -p 7005 CLUSTER MYID)
$ valkey-cli --cluster reshard 127.0.0.1:7000 --cluster-from $SRC --cluster-to $DST --cluster-slots 1 --cluster-yes --cluster-use-atomic-slot-migration | tail -5
       slots:[1197],[5461-10922] (5463 slots) master
       1 additional replica(s)
  Resharding plan:
    Moving slot range 0-0 from f1e432f739a537dc459403c56b61eeaf10e78aac
Moving slot range 0-0 from 127.0.0.1:7004 to 127.0.0.1:7005 via atomic slot migration.#
127.0.0.1:7000> PCACHE.GET 06S question
-> Redirected to slot [0] located at 127.0.0.1:7005
"answer"
```

**`redis-cli --cluster reshard`** moves keys with `MIGRATE`, one at a time.
The entries move when the source is told the slot is migrating (the first step
of the manual procedure below), and the tool needs no change:

```text
$ redis-cli -p 7000 CLUSTER NODES | awk "/master/ {print \$2, \$9, \$10, \$11, \$12}" | sort
127.0.0.1:7003@17003 10923-16229 16231-16383
127.0.0.1:7004@17004 1-1196 1198-5460
127.0.0.1:7005@17005 0 1197 5461-10922 16230
$ SRC=$(redis-cli -p 7005 CLUSTER MYID); DST=$(redis-cli -p 7004 CLUSTER MYID)
$ redis-cli --cluster reshard 127.0.0.1:7000 --cluster-from $SRC --cluster-to $DST --cluster-slots 1 --cluster-yes | tail -3
  Resharding plan:
    Moving slot 0 from 51ee6626e7f80308c24fd9ceb9b556b6c3a9dcd7
Moving slot 0 from 127.0.0.1:7005 to 127.0.0.1:7004:
127.0.0.1:7000> PCACHE.GET 06S question
-> Redirected to slot [0] located at 127.0.0.1:7004
"answer"
```

**Step by step with `CLUSTER SETSLOT`.** `SETSLOT MIGRATING` on the source
moves the entries of the slot to the target before it returns. While the slot is open, a command for an
entry that has moved gets `ASK`, and the target serves it after `ASKING`:

```text
$ SRC=$(redis-cli -p 7003 CLUSTER MYID); DST=$(redis-cli -p 7005 CLUSTER MYID)
$ redis-cli -p 7005 CLUSTER SETSLOT 16230 IMPORTING $SRC
OK
$ redis-cli -p 7003 CLUSTER SETSLOT 16230 MIGRATING $DST
OK
127.0.0.1:7003> SCACHE.GET faq 0.1 0.9 0.2
(error) ASK 16230 127.0.0.1:7005
127.0.0.1:7005> SCACHE.GET faq 0.1 0.9 0.2
(error) MOVED 16230 127.0.0.1:7003
127.0.0.1:7005> ASKING
OK
127.0.0.1:7005> SCACHE.GET faq 0.1 0.9 0.2
1) "Use the account page"
2) "1"
$ redis-cli -p 7005 CLUSTER SETSLOT 16230 NODE $DST; redis-cli -p 7003 CLUSTER SETSLOT 16230 NODE $DST
OK
OK
127.0.0.1:7000> SCACHE.GET faq 0.1 0.9 0.2
-> Redirected to slot [16230] located at 127.0.0.1:7005
1) "Use the account page"
2) "1"
```

If the target cannot take every entry of a slot, `SETSLOT MIGRATING` fails
and the slot stays where it was, so you can retry. The nodes use the cluster
password to talk to each other during a reshard: give every node the same
`requirepass`, and the same `masterauth` for the atomic migration, as
described in [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md).

The entries arrive complete: a semantic cache entry can still be found by its
embedding on the new owner.

## Next steps

- [Keys, slots and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md): hash tags and what
  answers per node.
- [Cluster clients](https://docs.hotkv.com/clustering/clients/index.md): the HotKV client libraries route
  the AI commands for you.
- [Scaling a cluster](https://docs.hotkv.com/clustering/scaling/index.md): reshard methods in full.