Replication

Replication keeps live copies of a primary’s data on one or more replicas. Replicas serve reads, give you a warm standby to promote if the primary is lost, and in cluster mode take over automatically when their primary fails.

Replication in HotKV is asynchronous, like Redis: a write is acknowledged to the client before replicas have it, and WAIT lets a client wait for replicas when it needs to.

How HotKV differs from Redis

The behaviour clients see (read-only replicas, ROLE, INFO replication, WAIT, WAITAOF, LOADING during a full sync) and the way you set a replica up (replicaof, REPLICAOF host port, min-replicas-to-write) follow Redis. The differences:

• A HotKV replica follows a HotKV primary only. Replication runs over the primary’s client port, with a protocol of its own. A Redis or Valkey replica cannot follow a HotKV primary, and a HotKV replica cannot follow a Redis primary: the link never comes up. Tools that rely on SYNC or PSYNC over the client port (for example redis-cli --rdb or --replica) do not work against HotKV.
• No automatic failover outside cluster mode, and no Sentinel. See High availability.
• Redis’s FAILOVER command is not supported. It replies ERR FAILOVER requires a running replica; replication is not configured. Promote a replica with REPLICAOF NO ONE.
• No extra replication port. Replicas connect to the primary’s client port. A separate replication port exists only if you set replication.dedicated-port yes (see Settings).

Setting up a primary and a replica

Start both nodes as ordinary HotKV servers. Then tell the replica where its primary is, at runtime with REPLICAOF:

127.0.0.1:8601> REPLICAOF 127.0.0.1 8600
OK
127.0.0.1:8600> SET greeting hello
OK
127.0.0.1:8600> WAIT 1 500
(integer) 1
127.0.0.1:8601> GET greeting
"hello"

The replica connects to the primary’s client port, copies its data, and then follows every write. Make it permanent with replicaof in the replica’s configuration file, or on the command line:

$ cat node-8602.conf
port 8602
dir node-8602
replicaof 127.0.0.1 8600
daemonize yes
logfile hotkv.log
$ redis-cli -p 8602 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
$ hotkv --port 8603 --dir node-8603 --daemonize yes --logfile hotkv.log --replicaof "127.0.0.1 8600"
$ redis-cli -p 8603 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
$ redis-cli -p 8600 INFO replication | grep -E "^(connected_slaves)"
connected_slaves:2

Check it with ROLE or INFO replication on either side:

127.0.0.1:8600> ROLE
1) "master"
2) (integer) 38
3) 1) 1) "127.0.0.1"
      2) "8601"
      3) "38"
127.0.0.1:8601> ROLE
1) "slave"
2) "127.0.0.1"
3) (integer) 8600
4) "connected"
5) (integer) 38
$ redis-cli -p 8600 INFO replication | grep -E "^(role|connected_slaves|slave0|master_repl_offset)"
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=8601,state=online,offset=38,lag=0
master_repl_offset:38
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status|master_sync_in_progress|slave_repl_offset|slave_read_only)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up
master_sync_in_progress:0
slave_repl_offset:38
slave_read_only:1

The primary lists its replicas with their offset and lag in seconds. The replica is caught up when its slave_repl_offset equals the primary’s master_repl_offset.

Authentication

Replicas and primaries authenticate each other with a shared replication secret. It is replication.password, or else masterauth, or else requirepass, so nodes that share the same requirepass need nothing more: a replica of a primary with requirepass pw, itself started with requirepass pw and no masterauth, connected at once.

A primary whose clients must authenticate but that has no replication secret refuses every replica. A node with no password at all serves replicas the way protected mode serves clients: only from loopback addresses while protected-mode is yes.

Settings

These are read at startup, except where CONFIG SET is listed.

Setting	Default	What it does
replicaof (slaveof), or REPLICAOF at runtime	none	The primary this node follows, as host port
replica-read-only (CONFIG SET too)	yes	Refuse writes from a replica’s own clients
masterauth (CONFIG SET too)	none	The password a replica uses to log in to its primary
min-replicas-to-write, min-replicas-max-lag (CONFIG SET too)	0, 10	Refuse writes on a primary unless this many replicas acknowledged within this many seconds
repl-backlog-size, repl-timeout, repl-ping-replica-period, repl-backlog-ttl (CONFIG SET too)	see the configuration reference	Backlog for partial resync, link timeout, heartbeat period, how long a primary without replicas keeps the backlog
shutdown-timeout (CONFIG SET too)	10	Seconds a primary waits for its replicas when it stops
replication.dedicated-port	no	Open a separate replication port; see below
tls-replication	no	Put every replication link on mutual TLS

The backlog counts toward maxmemory as it fills, up to its size. Its default is 64 MiB, or a sixteenth of maxmemory when that is less and maxmemory is set at start with no size given. INFO replication shows the size in force as repl_backlog_size. See Memory and eviction.

hotkv.conf also takes the replication.* settings (replication.enabled, replication.role, replication.primary-addr, replication.password, replication.port, replication.bind, replication.backlog-size, replication.replica-buffer-limit, replication.heartbeat-interval, and the [replication] section of hotkv.toml); the configuration reference lists them. A replica configured with replication.enabled yes, replication.role replica and replication.primary-addr host:port follows the primary at that client port:

$ cat node-8601.conf
port 8601
dir node-8601
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role replica
replication.primary-addr 127.0.0.1:8600
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up

Dedicated replication port. With replication.dedicated-port yes on the primary and on its replicas, the primary also listens on its own replication port (replication.port) and the replicas use it. Both the primary and its replicas need the setting, and replication.primary-addr of the replicas names that port. A replica without the setting that is pointed at the primary’s replication port never connects:

$ cat node-8600.conf
port 8600
dir node-8600
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role primary
replication.port 28600
replication.dedicated-port yes
$ cat node-8601.conf
port 8601
dir node-8601
daemonize yes
logfile hotkv.log
replication.enabled yes
replication.role replica
replication.primary-addr 127.0.0.1:28600
replication.dedicated-port yes
$ redis-cli -p 8601 INFO replication | grep -E "^(role|master_host|master_port|master_link_status)"
role:slave
master_host:127.0.0.1
master_port:8600
master_link_status:up

Caution

The replication port listens on every interface by default. Keep it reachable only from your other nodes, with a firewall or a private network, or set replication.bind. Without replication.dedicated-port no such port is open.

Over TLS

tls-replication yes puts every replication link on mutual TLS, using the node’s certificate settings. It is a restart-only setting, and both ends must have it, so turn it on for a primary and all its replicas together. See TLS.

Read-only replicas

A replica’s clients can read, but writes get an error, as in Redis:

127.0.0.1:8601> SET greeting hi
(error) READONLY You can't write against a read only replica.

replica-read-only no (read_only = false in TOML) lets a replica’s clients write locally. Those writes are not sent anywhere and the next full sync replaces them. CONFIG SET replica-read-only changes it at runtime.

A replica never evicts keys on its own when it passes maxmemory: its data is its primary’s, and the primary’s evictions reach it as deletes.

Full sync and partial resync

Full sync. When a replica connects for the first time, the primary sends it a complete copy of its data and then streams every write made since the copy was taken. The replica empties itself first. Until the copy has fully arrived, commands that read or write data on the replica get:

127.0.0.1:8601> GET k1
(error) LOADING HotKV is loading the dataset in memory
127.0.0.1:8601> PING
PONG
$ redis-cli -p 8601 INFO replication | grep -E "^(master_link_status|master_sync_in_progress)"
master_link_status:down
master_sync_in_progress:1

PING, INFO, AUTH, HELLO, CLIENT, CONFIG and ROLE answer, so monitoring keeps working. The primary keeps serving its own clients throughout. When the copy is done, the link is up and the replica serves its data.

Partial resync. A replica that reconnects after a short break (a network blip, a restart of its replication link) continues from its replication offset, and the primary sends only the writes it missed from its backlog, with no full sync and no LOADING. A full sync happens again when the backlog no longer holds those writes, when the primary process was restarted, or when the replica’s data cannot be trusted (a sync that was cut short, for example). After a promotion, the other replicas and the old primary can continue from the new primary without a full sync.

INFO stats on the primary counts full syncs (sync_full), granted partial resyncs (sync_partial_ok) and refused ones (sync_partial_err). The first sync of a replica counts as a refused partial resync followed by a full one, and a broken link that is resumed counts as a granted one:

$ redis-cli -p 8600 INFO stats | grep -E "^sync_(full|partial_ok|partial_err)"
sync_full:1
sync_partial_ok:0
sync_partial_err:1
127.0.0.1:8601> CLIENT KILL TYPE master
(integer) 1
$ redis-cli -p 8600 INFO stats | grep -E "^sync_(full|partial_ok|partial_err)"
sync_full:1
sync_partial_ok:1
sync_partial_err:1

Waiting for replicas

WAIT numreplicas timeout blocks until numreplicas replicas have applied every write this client made, or timeout milliseconds pass (0 waits indefinitely), and returns how many did:

127.0.0.1:8600> SET order:1001 paid
OK
127.0.0.1:8600> WAIT 1 1000
(integer) 1

WAITAOF numlocal numreplicas timeout waits until the writes are in this node’s AOF on disk (numlocal 1, which needs appendonly yes) and in the AOF on disk of numreplicas replicas. It returns two integers: whether the local AOF has them, and how many replicas’ AOFs do. A replica without an AOF never counts.

Two differences from Redis: WAIT on a replica returns 0 instead of an error, and a replica still completing its first full sync is not counted until it has acknowledged the write. WAITAOF on a replica returns an error, as in Redis.

WAIT does not make replication synchronous: a write that WAIT reports as not yet replicated has still been applied on the primary.

Refusing writes without replicas

min-replicas-to-write N makes a primary refuse writes unless at least N replicas have acknowledged within min-replicas-max-lag seconds. When fewer than N replicas have acknowledged in time, the primary refuses writes, and it accepts them again when enough replicas are back:

127.0.0.1:8600> CONFIG SET min-replicas-to-write 2
OK
127.0.0.1:8600> CONFIG SET min-replicas-max-lag 3
OK
127.0.0.1:8600> SET k 2
(error) NOREPLICAS Not enough good replicas to write.

Reads still work. INFO replication shows the number of replicas that count as good in min_slaves_good_slaves.

Stopping a primary

A primary that is stopped waits, up to shutdown-timeout seconds (default 10), until every replica has acknowledged every write, so a planned stop loses nothing. SHUTDOWN NOW skips the wait; see SHUTDOWN.

Promoting a replica

Outside cluster mode, promote a replica yourself with REPLICAOF NO ONE or REPLICATION.FAILOVER. It stops following its primary and accepts writes:

127.0.0.1:8601> REPLICAOF NO ONE
OK
127.0.0.1:8601> SET greeting promoted
OK
127.0.0.1:8601> ROLE
1) "master"
2) (integer) 79
3) (empty array)

Then:

1. Point your clients at the new primary.
2. Point the other replicas at it (REPLICAOF host port, and change replicaof in their files).
3. Change the promoted node’s own configuration so a restart does not turn it back into a replica: remove its replicaof line.
4. Bring the old primary back as a replica of the new one, not as a second primary.

To hand over without losing a write while the primary is still up: run CLIENT PAUSE 30000 WRITE on the primary (reads continue), wait until the replica’s slave_repl_offset in INFO replication equals the primary’s master_repl_offset, promote the replica, move clients to it, then run CLIENT UNPAUSE on the old primary.

Redis’s FAILOVER command is not supported: it replies with an error (ERR FAILOVER requires a running replica; replication is not configured). In cluster mode use CLUSTER FAILOVER instead; see Replicas and failover.

Replication in cluster mode

A cluster node needs no replication settings: a replica connects to its primary’s client port, and you attach one with redis-cli --cluster create ... --cluster-replicas 1, redis-cli --cluster add-node --cluster-slave, or CLUSTER REPLICATE. REPLICAOF is refused in cluster mode. Failover is automatic. See Replicas and failover and Clustering.

Enterprise replication commands

Cloud: ScaleOn-Premise: replication license

Command	What it does
REPLICATION.INFO	This node’s role, link state, replication offset and number of connected replicas
REPLICATION.PAUSE	Stop sending writes (on a primary) or applying them (on a replica)
REPLICATION.RESUME	Carry on from where the pause stopped
REPLICATION.FAILOVER	Promote this replica to primary (outside cluster mode)

127.0.0.1:8601> REPLICATION.INFO
1) "role"
2) "replica"
3) "state"
4) "streaming"
5) "repl_offset"
6) (integer) 0
7) "connected_replicas"
8) (integer) 0
127.0.0.1:8601> REPLICATION.PAUSE
OK
127.0.0.1:8600> SET p 1
OK
127.0.0.1:8601> GET p
(nil)
127.0.0.1:8601> REPLICATION.RESUME
OK
127.0.0.1:8601> GET p
"1"
127.0.0.1:8601> REPLICATION.FAILOVER
OK
127.0.0.1:8601> ROLE
1) "master"
2) (integer) 27
3) (empty array)
127.0.0.1:8600> REPLICATION.FAILOVER
(error) ERR This node is not a replica

While a primary is paused, its writes wait in the backlog and go out when it resumes; a replica that falls out of the backlog meanwhile resyncs in full. A paused replica stops answering its primary, which drops the link after a few heartbeat intervals; on resume the replica continues from the backlog.

Replication itself, WAIT, WAITAOF and REPLICAOF need no license; only the REPLICATION.* commands do. Without the license they reply with a NOLICENSE error:

127.0.0.1:8600> REPLICATION.INFO
(error) NOLICENSE replication requires a HotKV license. Contact sales@hotkv.com

HotKV Cloud

HotKV Cloud instances are single nodes: the panel does not provision replicas, so there is nothing to configure here. The Scale plan’s license includes the replication feature; on a single instance the REPLICATION.* commands report a primary with no connected replicas.

Next steps

• High availability
• Replicas and failover
• Persistence
• TLS