# Replication

> Set up HotKV primary-replica replication with REPLICAOF, read-only replicas, full and partial resync, WAIT and WAITAOF, min-replicas-to-write, and manual promotion.

URL: https://docs.hotkv.com/operate/replication/


Replication keeps live copies of a primary's data on one or more replicas.
Replicas serve reads, give you a warm standby to promote if the primary is
lost, and in [cluster mode](https://docs.hotkv.com/clustering/index.md) take over automatically when
their primary fails.

Replication in HotKV is asynchronous, like Redis: a write is acknowledged
to the client before replicas have it, and [`WAIT`](https://docs.hotkv.com/commands/wait/index.md) lets a
client wait for replicas when it needs to.

## How HotKV differs from Redis

The behaviour clients see (read-only replicas, `ROLE`, `INFO replication`,
`WAIT`, `WAITAOF`, `LOADING` during a full sync) and the way you set a replica up
(`replicaof`, `REPLICAOF host port`, `min-replicas-to-write`) follow Redis. The
differences:

- **A HotKV replica follows a HotKV primary only.** Replication runs over the
  primary's client port, with a protocol of its own. A Redis or Valkey replica cannot
  follow a HotKV primary, and a HotKV replica cannot follow a Redis primary: the
  link never comes up. Tools that rely on `SYNC` or `PSYNC` over the client port (for
  example `redis-cli --rdb` or `--replica`) do not work against HotKV.
- **No automatic failover outside cluster mode**, and no Sentinel. See
  [High availability](https://docs.hotkv.com/operate/high-availability/index.md).
- **Redis's `FAILOVER` command is not supported.** It replies `ERR FAILOVER requires a running
  replica; replication is not configured`. Promote a replica with `REPLICAOF NO ONE`.
- **No extra replication port.** Replicas connect to the primary's client port. A separate
  replication port exists only if you set `replication.dedicated-port yes` (see
  [Settings](#settings)).

## Setting up a primary and a replica

Start both nodes as ordinary HotKV servers. Then tell the replica where its
primary is, at runtime with [`REPLICAOF`](https://docs.hotkv.com/commands/replicaof/index.md):

```
127.0.0.1:8601> REPLICAOF 127.0.0.1 8600
OK
127.0.0.1:8600> SET greeting hello
OK
127.0.0.1:8600> WAIT 1 500
(integer) 1
127.0.0.1:8601> GET greeting
"hello"
```

The replica connects to the primary's client port, copies its data, and then follows every
write. Make it permanent with `replicaof` in the replica's configuration file, or on the
command line:

```
$ cat node-8602.conf
port 8602
dir node-8602
replicaof 127.0.0.1 8600
daemonize yes
logfile hotkv.log
$ redis-cli -p 8602 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
$ hotkv --port 8603 --dir node-8603 --daemonize yes --logfile hotkv.log --replicaof "127.0.0.1 8600"
$ redis-cli -p 8603 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
$ redis-cli -p 8600 INFO replication | grep -E "^(connected_slaves)"
connected_slaves:2
```

Check it with [`ROLE`](https://docs.hotkv.com/commands/role/index.md) or `INFO replication` on either side:

```
127.0.0.1:8600> ROLE
1) "master"
2) (integer) 38
3) 1) 1) "127.0.0.1"
      2) "8601"
      3) "38"
127.0.0.1:8601> ROLE
1) "slave"
2) "127.0.0.1"
3) (integer) 8600
4) "connected"
5) (integer) 38
$ redis-cli -p 8600 INFO replication | grep -E "^(role|connected_slaves|slave0|master_repl_offset)"
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=8601,state=online,offset=38,lag=0
master_repl_offset:38
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status|master_sync_in_progress|slave_repl_offset|slave_read_only)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
master_sync_in_progress:0
slave_repl_offset:38
slave_read_only:1
```

The primary lists its replicas with their offset and lag in seconds. The replica is caught up
when its `slave_repl_offset` equals the primary's `master_repl_offset`.

### Authentication

Replicas and primaries authenticate each other with a shared replication
secret. It is `replication.password`, or else `masterauth`, or else `requirepass`, so
nodes that share the same `requirepass` need nothing more: a replica of a primary with
`requirepass pw`, itself started with `requirepass pw` and no `masterauth`, connected at once.

A primary whose clients must authenticate but that has no replication
secret refuses every replica. A node with no password at all serves
replicas the way [protected mode](https://docs.hotkv.com/security/index.md#protected-mode) serves
clients: only from loopback addresses while `protected-mode` is `yes`.

### Settings

These are read at startup, except where `CONFIG SET` is listed.

| Setting | Default | What it does |
| --- | --- | --- |
| `replicaof` (`slaveof`), or `REPLICAOF` at runtime | none | The primary this node follows, as `host port` |
| `replica-read-only` (`CONFIG SET` too) | `yes` | Refuse writes from a replica's own clients |
| `masterauth` (`CONFIG SET` too) | none | The password a replica uses to log in to its primary |
| `min-replicas-to-write`, `min-replicas-max-lag` (`CONFIG SET` too) | `0`, `10` | Refuse writes on a primary unless this many replicas acknowledged within this many seconds |
| `repl-backlog-size`, `repl-timeout`, `repl-ping-replica-period`, `repl-backlog-ttl` (`CONFIG SET` too) | see the [configuration reference](https://docs.hotkv.com/reference/configuration/index.md#repl-backlog-size) | Backlog for partial resync, link timeout, heartbeat period, how long a primary without replicas keeps the backlog |
| `shutdown-timeout` (`CONFIG SET` too) | `10` | Seconds a primary waits for its replicas when it stops |
| `replication.dedicated-port` | `no` | Open a separate replication port; see below |
| `tls-replication` | `no` | Put every replication link on mutual TLS |

The backlog counts toward `maxmemory` as it fills, up to its size. Its default is
64 MiB, or a sixteenth of `maxmemory` when that is less and `maxmemory` is set at
start with no size given. `INFO replication` shows the size in force as
`repl_backlog_size`. See [Memory and eviction](https://docs.hotkv.com/operate/memory-and-eviction/index.md#the-replication-backlog).

`hotkv.conf` also takes the `replication.*` settings (`replication.enabled`,
`replication.role`, `replication.primary-addr`, `replication.password`, `replication.port`,
`replication.bind`, `replication.backlog-size`, `replication.replica-buffer-limit`,
`replication.heartbeat-interval`, and the `[replication]` section of `hotkv.toml`); the
[configuration reference](https://docs.hotkv.com/reference/configuration/index.md) lists them. A replica configured with
`replication.enabled yes`, `replication.role replica` and `replication.primary-addr host:port`
follows the primary at that **client port**:

```
$ cat node-8601.conf
port 8601
dir node-8601
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role replica
replication.primary-addr 127.0.0.1:8600
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
```

**Dedicated replication port.** With `replication.dedicated-port yes` on the primary and on its
replicas, the primary also listens on its own replication port (`replication.port`) and the replicas
use it. Both the primary and its replicas need the setting, and `replication.primary-addr` of
the replicas names that port. A replica without the setting that is pointed at the primary's replication
port never connects:

```
$ cat node-8600.conf
port 8600
dir node-8600
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role primary
replication.port 28600
replication.dedicated-port yes
$ cat node-8601.conf
port 8601
dir node-8601
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role replica
replication.primary-addr 127.0.0.1:28600
replication.dedicated-port yes
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
```

> **Caution**
> The replication port listens on every interface by default. Keep it reachable only from
> your other nodes, with a firewall or a private network, or set `replication.bind`. Without
> `replication.dedicated-port` no such port is open.

### Over TLS

`tls-replication yes` puts every replication link on mutual TLS, using the
node's certificate settings. It is a restart-only setting, and both ends
must have it, so turn it on for a primary and all its replicas together.
See [TLS](https://docs.hotkv.com/operate/tls/index.md).

## Read-only replicas

A replica's clients can read, but writes get an error, as in Redis:

```
127.0.0.1:8601> SET greeting hi
(error) READONLY You can't write against a read only replica.
```

`replica-read-only no` (`read_only = false` in TOML) lets a replica's
clients write locally. Those writes are not sent anywhere and the next full
sync replaces them. `CONFIG SET replica-read-only` changes it at runtime.

A replica never evicts keys on its own when it passes `maxmemory`: its data
is its primary's, and the primary's evictions reach it as deletes.

## Full sync and partial resync

**Full sync.** When a replica connects for the first time, the primary
sends it a complete copy of its data and then streams every write made
since the copy was taken. The replica empties itself first. Until the copy
has fully arrived, commands that read or write data on the replica get:

```
127.0.0.1:8601> GET k1
(error) LOADING HotKV is loading the dataset in memory
127.0.0.1:8601> PING
PONG
$ redis-cli -p 8601 INFO replication | grep -E "^(master_link_status|master_sync_in_progress)"
master_link_status:down
master_sync_in_progress:1
```

`PING`, `INFO`, `AUTH`, `HELLO`, `CLIENT`, `CONFIG` and `ROLE` answer, so monitoring keeps
working. The primary keeps serving its own clients throughout. When the copy is done, the link is up
and the replica serves its data.

**Partial resync.** A replica that reconnects after a short break (a
network blip, a restart of its replication link) continues from its
replication offset, and the primary sends only the writes it missed from
its backlog, with no full sync and no `LOADING`. A full sync happens again
when the backlog no longer holds those writes, when the primary process was
restarted, or when the replica's data cannot be trusted (a sync that was
cut short, for example). After a promotion, the other replicas and the old
primary can continue from the new primary without a full sync.

`INFO stats` on the primary counts full syncs (`sync_full`), granted
partial resyncs (`sync_partial_ok`) and refused ones (`sync_partial_err`). The first
sync of a replica counts as a refused partial resync followed by a full one, and a
broken link that is resumed counts as a granted one:

```
$ redis-cli -p 8600 INFO stats | grep -E "^sync_(full|partial_ok|partial_err)"
sync_full:1
sync_partial_ok:0
sync_partial_err:1
127.0.0.1:8601> CLIENT KILL TYPE master
(integer) 1
$ redis-cli -p 8600 INFO stats | grep -E "^sync_(full|partial_ok|partial_err)"
sync_full:1
sync_partial_ok:1
sync_partial_err:1
```

## Waiting for replicas

[`WAIT numreplicas timeout`](https://docs.hotkv.com/commands/wait/index.md) blocks until `numreplicas`
replicas have applied every write this client made, or `timeout`
milliseconds pass (`0` waits indefinitely), and returns how many did:

```
127.0.0.1:8600> SET order:1001 paid
OK
127.0.0.1:8600> WAIT 1 1000
(integer) 1
```

[`WAITAOF numlocal numreplicas timeout`](https://docs.hotkv.com/commands/waitaof/index.md) waits until the
writes are in this node's AOF on disk (`numlocal` 1, which needs
`appendonly yes`) and in the AOF on disk of `numreplicas` replicas. It
returns two integers: whether the local AOF has them, and how many replicas'
AOFs do. A replica without an AOF never counts.

Two differences from Redis: `WAIT` on a replica returns `0` instead of an
error, and a replica still completing its first full sync is not counted
until it has acknowledged the write. `WAITAOF` on a replica returns an
error, as in Redis.

`WAIT` does not make replication synchronous: a write that `WAIT` reports
as not yet replicated has still been applied on the primary.

## Refusing writes without replicas

`min-replicas-to-write N` makes a primary refuse writes unless at least `N` replicas
have acknowledged within `min-replicas-max-lag` seconds. When fewer than `N` replicas have
acknowledged in time, the primary refuses writes, and it accepts them again when enough
replicas are back:

```
127.0.0.1:8600> CONFIG SET min-replicas-to-write 2
OK
127.0.0.1:8600> CONFIG SET min-replicas-max-lag 3
OK
127.0.0.1:8600> SET k 2
(error) NOREPLICAS Not enough good replicas to write.
```

Reads still work. `INFO replication` shows the number of replicas that count as good in
`min_slaves_good_slaves`.

## Stopping a primary

A primary that is stopped waits, up to `shutdown-timeout` seconds (default 10), until every replica has
acknowledged every write, so a planned stop loses nothing. `SHUTDOWN NOW` skips the wait; see
[`SHUTDOWN`](https://docs.hotkv.com/commands/shutdown/index.md).

## Promoting a replica

Outside cluster mode, promote a replica yourself with
[`REPLICAOF NO ONE`](https://docs.hotkv.com/commands/replicaof/index.md) or
[`REPLICATION.FAILOVER`](https://docs.hotkv.com/commands/replication-failover/index.md). It stops
following its primary and accepts writes:

```
127.0.0.1:8601> REPLICAOF NO ONE
OK
127.0.0.1:8601> SET greeting promoted
OK
127.0.0.1:8601> ROLE
1) "master"
2) (integer) 79
3) (empty array)
```

Then:

1. Point your clients at the new primary.
2. Point the other replicas at it (`REPLICAOF host port`, and change `replicaof` in their
   files).
3. Change the promoted node's own configuration so a restart does not turn it back into
   a replica: remove its `replicaof` line.
4. Bring the old primary back as a replica of the new one, not as a second primary.

To hand over without losing a write while the primary is still up: run
`CLIENT PAUSE 30000 WRITE` on the primary (reads continue), wait until the
replica's `slave_repl_offset` in `INFO replication` equals the primary's
`master_repl_offset`, promote the replica, move clients to it, then run
`CLIENT UNPAUSE` on the old primary.

Redis's `FAILOVER` command is not supported: it replies with an error
(`ERR FAILOVER requires a running replica; replication is not configured`). In cluster mode use
[`CLUSTER FAILOVER`](https://docs.hotkv.com/commands/cluster-failover/index.md) instead; see
[Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md).

## Replication in cluster mode

A cluster node needs no replication settings: a replica connects to its primary's
client port, and you attach one with
`redis-cli --cluster create ... --cluster-replicas 1`,
`redis-cli --cluster add-node --cluster-slave`, or
[`CLUSTER REPLICATE`](https://docs.hotkv.com/commands/cluster-replicate/index.md). `REPLICAOF` is refused in cluster mode. Failover is
automatic. See [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md) and
[Clustering](https://docs.hotkv.com/clustering/index.md).

## Enterprise replication commands

**Availability:** HotKV Cloud: Scale plan. On-Premise: requires the `replication` license feature.

| Command | What it does |
| --- | --- |
| [`REPLICATION.INFO`](https://docs.hotkv.com/commands/replication-info/index.md) | This node's role, link state, replication offset and number of connected replicas |
| [`REPLICATION.PAUSE`](https://docs.hotkv.com/commands/replication-pause/index.md) | Stop sending writes (on a primary) or applying them (on a replica) |
| [`REPLICATION.RESUME`](https://docs.hotkv.com/commands/replication-resume/index.md) | Carry on from where the pause stopped |
| [`REPLICATION.FAILOVER`](https://docs.hotkv.com/commands/replication-failover/index.md) | Promote this replica to primary (outside cluster mode) |

```
127.0.0.1:8601> REPLICATION.INFO
1) "role"
2) "replica"
3) "state"
4) "streaming"
5) "repl_offset"
6) (integer) 0
7) "connected_replicas"
8) (integer) 0
127.0.0.1:8601> REPLICATION.PAUSE
OK
127.0.0.1:8600> SET p 1
OK
127.0.0.1:8601> GET p
(nil)
127.0.0.1:8601> REPLICATION.RESUME
OK
127.0.0.1:8601> GET p
"1"
127.0.0.1:8601> REPLICATION.FAILOVER
OK
127.0.0.1:8601> ROLE
1) "master"
2) (integer) 27
3) (empty array)
127.0.0.1:8600> REPLICATION.FAILOVER
(error) ERR This node is not a replica
```

While a primary is paused, its writes wait in the backlog and go out when it
resumes; a replica that falls out of the backlog meanwhile resyncs in full.
A paused replica stops answering its primary, which drops the link after a
few heartbeat intervals; on resume the replica continues from the backlog.

Replication itself, `WAIT`, `WAITAOF` and `REPLICAOF` need no license; only the
`REPLICATION.*` commands do. Without the license they reply with a `NOLICENSE` error:

```
127.0.0.1:8600> REPLICATION.INFO
(error) NOLICENSE replication requires a HotKV license. Contact sales@hotkv.com
```

## HotKV Cloud

HotKV Cloud instances are single nodes: the panel does not provision
replicas, so there is nothing to configure here. The Scale plan's license
includes the `replication` feature; on a single instance the
`REPLICATION.*` commands report a primary with no connected replicas.

## Next steps

- [High availability](https://docs.hotkv.com/operate/high-availability/index.md)
- [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md)
- [Persistence](https://docs.hotkv.com/persistence/index.md)
- [TLS](https://docs.hotkv.com/operate/tls/index.md)