# Clustering

> What a HotKV cluster is for, how many nodes you need, how slots, redirects and failover work, and where to find every cluster guide.

URL: https://docs.hotkv.com/clustering/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

A HotKV cluster splits your data over several nodes and keeps working when a
node fails. Every key belongs to one of 16384 hash slots, every slot belongs
to one primary node, and each primary can have replicas that take over when it
stops. Cluster mode follows the Redis Cluster protocol, so `redis-cli`,
`valkey-cli` and the cluster clients of Redis and Valkey libraries work
against it. The nodes of a HotKV cluster talk to each other over a bus protocol
of their own, so a HotKV node cannot join a Redis or Valkey cluster (see
[What does not work](https://docs.hotkv.com/clustering/migrate-to-hotkv/index.md#what-does-not-work)). It needs
no license. It runs On-Premise: HotKV Cloud instances are single nodes.

If you want to see one first, the [quick start](https://docs.hotkv.com/clustering/quick-start/index.md)
builds a six-node cluster on one machine in a few minutes.

## Do you need a cluster?

A single HotKV node is multi-threaded and by default uses every CPU core of
its machine ([`threads`](https://docs.hotkv.com/reference/configuration/index.md#threads)), so you do not
need a cluster to use a large machine. Choose a cluster when one of these is
true:

| You need | Why a cluster helps |
| --- | --- |
| More data than one node can hold | Slots, and the keys in them, are spread over the primaries, and you add primaries to grow. |
| Automatic failover | There is no Sentinel in HotKV. In a cluster, the replicas of a failed primary elect a new one without an operator. |
| More write capacity than one machine gives | Writes to different slots go to different primaries. |
| AI features beyond one node | Caches, vector indexes, budgets and the like are spread over the nodes by name (see [HotKV specifics](#hotkv-specifics)). |

Stay with one node, or one primary with replicas, when your data fits on a
machine and a short outage while you promote a replica yourself is
acceptable. That setup is simpler: no slots, no redirects, and multi-key
commands, transactions and scripts work across all your keys. See
[Replication](https://docs.hotkv.com/operate/replication/index.md) and
[High availability](https://docs.hotkv.com/operate/high-availability/index.md).

A cluster has costs. A command that names several keys needs them in the same
slot, `SELECT` and `MOVE` between databases are off by default, and a client
must follow redirects. [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md)
lists the rules.

## How a cluster works

- **Slots.** A key's slot is its CRC16 checksum modulo 16384. A cluster is
  complete when every slot has an owner.
- **Hash tags.** If a key contains `{tag}`, only the text between the first
  `{` and the next `}` is hashed. `{user:1}:name` and `{user:1}:city` share a
  slot, so one multi-key command can use both.
- **Shards.** A shard is a primary and its replicas. Replicas copy their
  primary's data, and a replica serves reads only after the client sends
  `READONLY`.
- **The cluster bus.** Nodes talk to each other on a second port, the bus,
  which is the client port plus 10000 (or `cluster-port`). They use it to
  gossip, to agree that a node has failed and to vote in elections. Replicas
  copy data over the primary's client port.
- **Asynchronous replication.** A primary acknowledges a write before its
  replicas have it, so a primary that dies can lose its latest writes. `WAIT`
  (replicas) and `WAITAOF` (disk) make a client wait for acknowledgements
  when that matters.
- **The majority rule.** A node is declared failed only when a majority of
  the primaries that own slots report it unreachable. A node that cannot
  reach a majority of those primaries stops serving and answers
  `CLUSTERDOWN`, so the smaller side of a network split never accepts writes
  the other side will override. With two primaries, losing one leaves the
  other without a majority, which is why a cluster needs three.
- **`nodes.conf`.** Each node writes the cluster as it knows it to this file
  in its data directory. It holds the node's own ID, so keep it with the data,
  never share it between nodes and do not edit it by hand.

## How big should it be?

### Nodes

| Question | Answer |
| --- | --- |
| How many primaries? | At least 3. `redis-cli --cluster create` refuses fewer than 3 primaries, and a two-primary cluster built by hand stops serving when either primary fails. Add primaries when the data or the write rate outgrows the ones you have. |
| How many replicas per primary? | At least 1, or a failed primary has nothing to take over. Use 2 when you must survive a second failure in the same shard before the first node is replaced. |
| Where do replicas go? | On a different host (and, in a cloud, a different availability zone) than their primary. When you pass several hosts to `redis-cli --cluster create`, it places replicas away from their primary. |
| How many nodes per host? | Ideally one. If you run several on one host, never put a primary and its replica together: the host would be a single point of failure. |
| How much memory? | Set [`maxmemory`](https://docs.hotkv.com/reference/configuration/index.md#maxmemory) below the memory of the machine and leave room beyond the data for replication buffers. A node that receives slots in a reshard needs room for them. |

A six-node cluster (3 primaries and 3 replicas) is the smallest layout that
survives the loss of any one node.

### Ports

| Port | What it is for | Who must reach it |
| --- | --- | --- |
| Client port, for example `7000` | Clients, `redis-cli --cluster`, and replicas copying from their primary | Clients and the other nodes |
| Bus port, client port + 10000, for example `17000` | Gossip, failure detection and elections | The other nodes only, never the internet |
| TLS port (`tls-port`), when you use TLS | The same as the client port, over TLS. With `tls-cluster yes` the bus port is the TLS port + 10000. | As above |

A dedicated replication port (client port + 20000, or the TLS port + 20000 with
`tls-cluster yes`) exists only when you set `replication.dedicated-port yes`.

### TLS or not

| Situation | Advice |
| --- | --- |
| All nodes and clients are on a private network you control | Plain text is fine. Keep the bus port reachable only from the nodes, and set a password. |
| Nodes or clients cross a network you do not control (the internet, another provider, a shared network) | Use TLS for clients and, with `tls-cluster` and `tls-replication`, for the bus and replication. See [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md). |

Keep the bus port reachable only from the nodes, and use `tls-cluster` on
networks you do not control. The
[production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md) covers both.

### Failure detection

[`cluster-node-timeout`](https://docs.hotkv.com/reference/configuration/index.md#cluster-node-timeout)
(default 15000 ms) is how long a node may stay silent before the others flag
it. Failover takes roughly the node timeout plus a few seconds. Lower it for
faster failover on a stable network. A very low value makes a short pause look
like a failure and causes failovers nobody needed. See
[Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md#failure-detection).

## Compatibility

A HotKV cluster covers the cluster features of Redis 8 and Valkey 9:

- Redis Cluster protocol: `MOVED`, `ASK`, `CLUSTER SLOTS`, `CLUSTER SHARDS`,
  `CLUSTER NODES`, `READONLY`, automatic failover and manual failover.
- Both atomic slot migration dialects: [`CLUSTER MIGRATESLOTS`](https://docs.hotkv.com/commands/cluster-migrateslots/index.md)
  (Valkey 9) and [`CLUSTER MIGRATION`](https://docs.hotkv.com/commands/cluster-migration/index.md) (Redis 8.4 and
  later), next to the classic `MIGRATE` based reshard.
- Several databases per node with `cluster-databases`.
- Hostnames in redirects (`cluster-announce-hostname` and
  `cluster-preferred-endpoint-type`), the client-facing address settings for NAT and
  load balancers, and `availability-zone`.
- [`CLUSTERSCAN`](https://docs.hotkv.com/commands/clusterscan/index.md), [`CLUSTER SLOT-STATS`](https://docs.hotkv.com/commands/cluster-slot-stats/index.md)
  and sharded pub/sub (`SSUBSCRIBE`, `SPUBLISH`).

Two things differ from what you may expect. `hotkv-cli` has no cluster mode,
so create and drive clusters with `redis-cli` or `valkey-cli`. And Sentinel
is not supported: failover in a cluster is built in.

## HotKV specifics

HotKV's AI and data features are clustered along with the keys. A cache
namespace, a vector index, a RAG pipeline, an agent's memory, a rate limiter,
a budget or a time series lives on the node that owns the slot of its name. A
reshard moves it with the slot, replicas copy it, and a promoted replica keeps
it. Give names that must stay together a common `{hash tag}`. Some state is
per node: the license, listings such as `PCACHE.LIST`, scripts, function
libraries and ACL users. Read these pages for the details:

- [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md): what is clustered, what
  cannot be, and the rules for multi-key commands.
- [AI engines in a cluster](https://docs.hotkv.com/clustering/ai-engines/index.md): sharding by name, what is
  replicated, global tables and licensing per node.
- [Clients](https://docs.hotkv.com/clustering/clients/index.md): cluster clients that follow redirects and
  survive failover.

> **Note**
> Cluster mode needs no license, but each node needs its own license for the
> AI features it serves.

## Slots and redirects

When a command's keys are not on the node that received it, the node answers
with an error that tells the client what to do. This table lists the replies
a cluster gives:

<div className="table-scroll">
<table>
<thead><tr><th>Reply</th><th>Meaning and what to do</th></tr></thead>
<tbody>
<tr><td><code>-MOVED 12714 127.0.0.1:7002</code></td><td>The slot belongs to another node. Retry the command there and update your slot map.</td></tr>
<tr><td><code>-ASK 15891 127.0.0.1:7200</code></td><td>The slot is being moved and this key is no longer here. Send <code>ASKING</code> to the named node, then the command. Do not update your slot map.</td></tr>
<tr><td><code>-CROSSSLOT Keys in request don't hash to the same slot</code></td><td>A multi-key command, transaction or script names keys in different slots. Use a hash tag so they share a slot.</td></tr>
<tr><td><code>-TRYAGAIN Multiple keys request during rehashing of slot</code></td><td>A multi-key command found some of its keys already moved and some not, while a slot is migrating. Retry shortly.</td></tr>
<tr><td><code>-CLUSTERDOWN The cluster is down</code></td><td>This node reports <code>cluster_state:fail</code>: a slot has no working owner and <code>cluster-require-full-coverage</code> is <code>yes</code>, or this node is on the minority side of a split.</td></tr>
<tr><td><code>-CLUSTERDOWN Hash slot not served</code></td><td>No node owns this key's slot, whatever the cluster state says.</td></tr>
<tr><td><code>-CLUSTERDOWN The cluster is down and only accepts read commands</code></td><td>The cluster is down and <code>cluster-allow-reads-when-down</code> is <code>yes</code>: reads work, writes get this.</td></tr>
</tbody>
</table>
</div>

A replica answers `MOVED` to a read until the client sends `READONLY`, and to
every write:

```text
127.0.0.1:7204> GET {t}2
(error) MOVED 15891 127.0.0.1:7202
127.0.0.1:7204> READONLY
OK
127.0.0.1:7204> GET {t}2
"two"
```

What the address in `MOVED` and `ASK` looks like depends on
`cluster-preferred-endpoint-type` and the announce settings, covered in
[Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md#announce-addresses-and-nat).
Cluster clients follow `MOVED` and `ASK` for you and refresh their slot map.

For the cause and fix of each reply and of each error you may meet while
setting up, see [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md).

## The guides

| Guide | What it covers |
| --- | --- |
| [Cluster quick start](https://docs.hotkv.com/clustering/quick-start/index.md) | A local six-node cluster, a failover demo and cleanup |
| [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md) | Sizing, systemd, ports and firewalls, authentication, announce settings, creating a cluster across hosts |
| [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md) | Certificates, configuration, clients, renewal |
| [Docker Compose](https://docs.hotkv.com/clustering/docker-compose/index.md) and [Kubernetes](https://docs.hotkv.com/clustering/kubernetes/index.md) | Running a cluster in containers |
| [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md) | Slots, hash tags and what a cluster can and cannot do |
| [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md) | How many replicas, reading from replicas, automatic and manual failover |
| [Scaling](https://docs.hotkv.com/clustering/scaling/index.md) | Adding and removing nodes, resharding |
| [AI engines](https://docs.hotkv.com/clustering/ai-engines/index.md) and [Clients](https://docs.hotkv.com/clustering/clients/index.md) | AI features and client libraries in a cluster |
| [Upgrades](https://docs.hotkv.com/clustering/upgrades/index.md) | Rolling upgrades |
| [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md) | Per-node backups, restoring nodes and clusters, moving to new hosts |
| [Monitoring](https://docs.hotkv.com/clustering/monitoring/index.md) | What to watch and alert on |
| [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md) | Symptoms, causes and fixes |
| [Production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md) | Everything to check before you go live |
| [Migrate to HotKV](https://docs.hotkv.com/clustering/migrate-to-hotkv/index.md) | Moving from a Redis or Valkey cluster |
| [Cluster configuration](https://docs.hotkv.com/clustering/configuration/index.md) | Every cluster and replication setting |