# Cluster over TLS

> Run a HotKV cluster with TLS on client connections, the cluster bus and replication, from creating certificates to renewing them without a restart.

URL: https://docs.hotkv.com/clustering/tls/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

TLS in a cluster covers three kinds of connection: your clients, the links
between nodes (the cluster bus), and the replication links from replicas to
their primaries. You turn each on separately. Between nodes HotKV always uses
mutual TLS: both ends present a certificate and each checks the other's
against your CA, so a host without a certificate from your CA cannot join the
cluster even when it can reach the bus port.

This page builds a TLS cluster across two hosts, from the certificates up.
The two hosts are `10.0.0.1` (`node1.example.com`) and `10.0.0.2`
(`node2.example.com`), each running three nodes on ports 7000 to 7002. The
TLS settings for a single server, outside a cluster, are on
[TLS](https://docs.hotkv.com/operate/tls/index.md). Read [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md)
first for the parts that are the same with and without TLS: sizing,
firewalls, passwords.

The commands below write under `/etc/hotkv`, `/var/lib/hotkv` and
`/var/log/hotkv` and read private keys, so run them in a root shell (`sudo -i`)
on each host.

## How the ports work

With `tls-cluster yes` a node goes by its TLS port in the cluster, and the bus
port follows it:

| Settings | Client ports | Bus port |
| --- | --- | --- |
| `port 0`, `tls-port 7000` (TLS only) | TLS on 7000 | 17000 |
| `port 7060`, `tls-port 7160` (both) | plain on 7060, TLS on 7160 | 17160 |

`cluster-port` puts the bus on a port of your choice instead. With both ports,
each client is told the port of its own kind. This is what a
node with `port 7060` and `tls-port 7160` printed for `CLUSTER NODES`, to a
plain client and to a TLS client:

```text
88c1a627187760c03ab057b49f2eb0c6b19121d8 10.0.0.1:7060@17160 myself,master - 0 0 0 connected
88c1a627187760c03ab057b49f2eb0c6b19121d8 10.0.0.1:7160@17160 myself,master - 0 0 0 connected
```

The bus was listening on 17160, the TLS port plus 10000, and nothing
listened on 17060. Replicas connect to the primary's TLS port. Open the TLS
port and the bus port between the nodes, as described in
[Ports and firewalls](https://docs.hotkv.com/clustering/production/index.md#ports-and-firewalls).

## 1. Create the certificates

You need one CA that signs only the certificates of this cluster, a
certificate for each host, and a certificate for your applications. Run these
commands in an empty directory on a trusted machine (here, the first host).
The certificates here use elliptic-curve keys and last a year; the CA lasts
ten.

```bash
mkdir pki && cd pki
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 -subj "/CN=HotKV cluster CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" -out ca.crt
for node in node1:10.0.0.1 node2:10.0.0.2; do
  name=${node%%:*}; ip=${node##*:}
  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out $name.key
  openssl req -new -key $name.key -subj "/CN=$name.example.com" -out $name.csr
  openssl x509 -req -in $name.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -days 365 \
    -extfile <(printf "subjectAltName=DNS:%s.example.com,IP:%s,IP:127.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\n" $name $ip) \
    -out $name.crt
done
ls
cd ..
```

```text
Certificate request self-signature ok
subject=CN=node1.example.com
Certificate request self-signature ok
subject=CN=node2.example.com
ca.crt
ca.key
ca.srl
node1.crt
node1.csr
node1.key
node2.crt
node2.csr
node2.key
```

Two things in the certificates matter:

- **Names.** The subject alternative name lists every address and host name
  that clients use to reach the node, including the IP addresses, and
  `127.0.0.1` for tools run on the node itself. Many client libraries check
  that the name they connected to is in the certificate. Between nodes the
  names are not checked unless you set `tls-expected-peer-name` (see
  [Bus protection](#bus-protection)).
- **Key usage.** Each node uses its certificate both to accept connections
  and to open them to other nodes, so it needs `serverAuth` and `clientAuth`.
  A certificate with `serverAuth` only is refused by the other nodes; in that
  case give the node a second certificate with `tls-client-cert-file` and
  `tls-client-key-file`.

```bash
(cd pki && openssl x509 -in node1.crt -noout -subject -issuer -ext subjectAltName,extendedKeyUsage -dates)
```

```text
subject=CN=node1.example.com
issuer=CN=HotKV cluster CA
X509v3 Subject Alternative Name:
    DNS:node1.example.com, IP Address:10.0.0.1, IP Address:127.0.0.1
X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication
notBefore=Jan  1 00:00:00 2027 GMT
notAfter=Jan  1 00:00:00 2028 GMT
```

Create a certificate for your applications the same way, with
`extendedKeyUsage=clientAuth`. You need it because `tls-auth-clients` is
`yes` by default, which makes every TLS client present a certificate:

```bash
(cd pki
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
openssl req -new -key app.key -subj "/CN=app" -out app.csr
openssl x509 -req -in app.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -days 365 \
  -extfile <(printf "extendedKeyUsage=clientAuth\n") -out app.crt
ls app.*)
```

```text
Certificate request self-signature ok
subject=CN=app
app.crt
app.csr
app.key
```

Install the CA certificate and the host's own certificate and key under
`/etc/hotkv/tls/` on each host (the directory does not exist yet), with the key
readable only by root and the `hotkv` group, the user the HotKV service runs as.
On the first host:

```bash
install -d -m 755 /etc/hotkv/tls
(cd pki && install -m 644 ca.crt /etc/hotkv/tls/ca.crt && install -m 644 node1.crt /etc/hotkv/tls/node.crt && install -m 640 -g hotkv node1.key /etc/hotkv/tls/node.key)
stat -c '%A %U:%G %s %n' /etc/hotkv/tls/*
```

```text
-rw-r--r-- root:root 615 /etc/hotkv/tls/ca.crt
-rw-r--r-- root:root 676 /etc/hotkv/tls/node.crt
-rw-r----- root:hotkv 241 /etc/hotkv/tls/node.key
```

On the second host, install `ca.crt`, `node2.crt` as `node.crt` and `node2.key`
as `node.key` the same way, copying them over a channel you trust. Never copy
`ca.key` to the nodes: it can sign certificates for anyone. Keep it offline
and use it only to issue and renew certificates. A node that runs as the
`hotkv` user, as the systemd units do, cannot start with a key that only root
can read: it logs `Failed to initialize SSL context: use_private_key_file: Permission denied` and exits.

## 2. Configure the nodes

The file for the first node on the first host, `/etc/hotkv/7000.conf`:

```text
bind 10.0.0.1 127.0.0.1
port 0
tls-port 7000
tls-cert-file /etc/hotkv/tls/node.crt
tls-key-file /etc/hotkv/tls/node.key
tls-ca-cert-file /etc/hotkv/tls/ca.crt
tls-auth-clients yes
tls-cluster yes
tls-replication yes
requirepass change-me
cluster-enabled yes
dir /var/lib/hotkv/7000
```

`port 0` turns the plain listener off, so the node serves TLS only.
`tls-cluster yes` puts the bus on TLS and `tls-replication yes` does the same
for replication. A node with either setting and no certificate, key and CA (`tls-ca-cert-file` or
`tls-ca-cert-dir`) does not start. The
password is the same one as without TLS: TLS protects the connection, and the
password and ACL users still decide who may do what.

The same node in `hotkv.toml`:

```toml
[network]
bind = "10.0.0.1 127.0.0.1"
port = 0

[tls]
port = 7000
cert_file = "/etc/hotkv/tls/node.crt"
key_file = "/etc/hotkv/tls/node.key"
ca_cert_file = "/etc/hotkv/tls/ca.crt"
auth_clients = "yes"
cluster = true
replication = true

[security]
requirepass = "change-me"

[cluster]
enabled = true

[persistence]
dir = "/var/lib/hotkv/7000"
```

Make the files for ports 7001 and 7002 from the first one, and the files for
the second host the same way with `bind 10.0.0.2 127.0.0.1`:

```bash
cat > /etc/hotkv/7000.conf <<'EOF'
bind 10.0.0.1 127.0.0.1
port 0
tls-port 7000
tls-cert-file /etc/hotkv/tls/node.crt
tls-key-file /etc/hotkv/tls/node.key
tls-ca-cert-file /etc/hotkv/tls/ca.crt
tls-auth-clients yes
tls-cluster yes
tls-replication yes
requirepass change-me
cluster-enabled yes
dir /var/lib/hotkv/7000
EOF
for port in 7001 7002; do sed s/7000/$port/g /etc/hotkv/7000.conf > /etc/hotkv/$port.conf; done
chgrp hotkv /etc/hotkv/700?.conf; chmod 640 /etc/hotkv/700?.conf
install -d -o hotkv -g hotkv /var/lib/hotkv/7000 /var/lib/hotkv/7001 /var/lib/hotkv/7002 /var/log/hotkv
cat /etc/hotkv/7001.conf
```

```text
bind 10.0.0.1 127.0.0.1
port 0
tls-port 7001
tls-cert-file /etc/hotkv/tls/node.crt
tls-key-file /etc/hotkv/tls/node.key
tls-ca-cert-file /etc/hotkv/tls/ca.crt
tls-auth-clients yes
tls-cluster yes
tls-replication yes
requirepass change-me
cluster-enabled yes
dir /var/lib/hotkv/7001
```

Start the nodes on both hosts. Under systemd, use the unit from
[Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md#run-nodes-under-systemd).
For a quick test, from a shell (the data directories were created above):

```bash
for port in 7000 7001 7002; do
  hotkv /etc/hotkv/$port.conf --daemonize yes --pidfile /var/lib/hotkv/$port/hotkv.pid \
    --logfile /var/log/hotkv/$port.log
done
```

The log of a node says what it set up:

```text
Plain-text port disabled (port 0), serving TLS only
SSL context initialized (cert=/etc/hotkv/tls/node.crt, key=/etc/hotkv/tls/node.key)
TLS on the cluster bus, other nodes verified against the CA
TLS on replication links, other nodes verified against the CA
TLS enabled on 10.0.0.1 127.0.0.1:7000
```

(Log lines here are shown without the date, time and level that start each
line.)

## 3. Create the cluster

Give `redis-cli` the CA certificate to verify the nodes, and a certificate
and key to present to them. The node's own certificate works, since it has
`clientAuth`. The password goes in the environment:

```bash
export REDISCLI_AUTH=change-me
redis-cli --tls --cacert /etc/hotkv/tls/ca.crt --cert /etc/hotkv/tls/node.crt --key /etc/hotkv/tls/node.key \
  --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 \
  10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes
```

```text
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.2:7002 to 10.0.0.1:7000
Adding replica 10.0.0.1:7002 to 10.0.0.2:7000
Adding replica 10.0.0.2:7001 to 10.0.0.1:7001
M: 4fa60fec899cdebbd189d9f7c623365ce7598a9d 10.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: ef33b047c890fdd8fa707eb7a7e5e51149723ab1 10.0.0.1:7001
   slots:[10923-16383] (5461 slots) master
S: 3a6ba9f419bdc912c80632d823cc8f77e3f2ffe5 10.0.0.1:7002
   replicates 8a58921d8b2ad80e67568d708d701ea0cf6c0fef
M: 8a58921d8b2ad80e67568d708d701ea0cf6c0fef 10.0.0.2:7000
   slots:[5461-10922] (5462 slots) master
S: c61c540db15f2c19f8ff003a31d25762a18d9338 10.0.0.2:7001
   replicates ef33b047c890fdd8fa707eb7a7e5e51149723ab1
S: 2a8484d5defce55e2bb8ecebc82b69c85897b811 10.0.0.2:7002
   replicates 4fa60fec899cdebbd189d9f7c623365ce7598a9d
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 10.0.0.1:7000)
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

`valkey-cli` takes the same options (and `VALKEYCLI_AUTH` for the password).
Starting from fresh nodes, its create ends the same way:

```bash
export VALKEYCLI_AUTH=change-me
valkey-cli --tls --cacert /etc/hotkv/tls/ca.crt --cert /etc/hotkv/tls/node.crt --key /etc/hotkv/tls/node.key \
  --cluster create 10.0.0.1:7000 10.0.0.1:7001 10.0.0.1:7002 \
  10.0.0.2:7000 10.0.0.2:7001 10.0.0.2:7002 --cluster-replicas 1 --cluster-yes | tail -n 12
```

```text
   slots: (0 slots) slave
   replicates fa81bb8a7759a36922313e397f1ee40c7423d2fa
S: 9a0f41a11ce7e331604025cc2d00fcad2a41b9de 10.0.0.1:7002
   slots: (0 slots) slave
   replicates 10dfd88397579c4244058f8ec8526936fc6d5f67
M: fa81bb8a7759a36922313e397f1ee40c7423d2fa 10.0.0.1:7001
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
```

## 4. Check that everything is encrypted

A plain connection to a client port or the bus port of a TLS-only node is
closed:

```bash
redis-cli -h 10.0.0.1 -p 7000 PING
redis-cli -h 10.0.0.1 -p 17000 PING
```

```text
Error: Server closed the connection
Error: Server closed the connection
```

The node logs each refused connection, and says what it was:

```text
TLS handshake failed for client 4: wrong version number (SSL routines)
Error accepting cluster node connection: wrong version number (SSL routines) (the TLS handshake from 10.0.0.1: tls-cluster is on here, the other nodes must use it too, with a certificate tls-ca-cert-file trusts)
```

A TLS client that presents no certificate is refused too, while
`tls-auth-clients` is `yes`:

```text
TLS handshake failed for client 5: peer did not return a certificate (SSL routines)
```

The bus port speaks TLS. `openssl` can connect to it with a node certificate
and shows the protocol and the peer's certificate (the long line of signature
algorithms is cut here):

```bash
openssl s_client -connect 10.0.0.1:17000 -CAfile /etc/hotkv/tls/ca.crt -cert /etc/hotkv/tls/node.crt -key /etc/hotkv/tls/node.key -verify_return_error -brief </dev/null 2>&1 | head -n 8
```

```text
Connecting to 10.0.0.1
Can't use SSL_get_servername
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
...
Peer certificate: CN=node1.example.com
Hash used: SHA256
```

Replication is encrypted too. A replica's log shows it, and `INFO replication`
shows the link is up:

```text
Replication: becoming a replica of 10.0.0.2:7000
Replication: connecting to primary 10.0.0.2:7000
Replication: connected to primary 10.0.0.2:7000 over TLS
```

```bash
redis-cli --tls --cacert /etc/hotkv/tls/ca.crt --cert /etc/hotkv/tls/node.crt --key /etc/hotkv/tls/node.key -p 7002 INFO replication | grep -E "role|master_link_status|master_port"
```

```text
role:slave
master_port:7000
master_link_status:up
```

`CLUSTER LINKS` lists the bus links of a node with their buffers, but does
not say whether they are encrypted. Use the log lines above for that.

## 5. Connect clients

`redis-cli` with the application certificate follows redirects as usual:

```bash
REDISCLI_AUTH=change-me redis-cli -c --tls --cacert pki/ca.crt --cert pki/app.crt --key pki/app.key -h 10.0.0.1 -p 7000 SET greeting hello
REDISCLI_AUTH=change-me redis-cli -c --tls --cacert pki/ca.crt --cert pki/app.crt --key pki/app.key -h 10.0.0.2 -p 7002 GET greeting
```

```text
OK
hello
```

In Python, the `redis` package (`pip install redis`, in a virtual environment as
shown in [Cluster clients](https://docs.hotkv.com/clustering/clients/index.md)) takes the certificates as
arguments. This script works against the cluster above:

```python
from redis.cluster import RedisCluster

rc = RedisCluster(
    host="10.0.0.1",
    port=7000,
    password="change-me",
    ssl=True,
    ssl_ca_certs="pki/ca.crt",
    ssl_certfile="pki/app.crt",
    ssl_keyfile="pki/app.key",
)
rc.set("greeting", "hello from python")
print(rc.get("greeting"))
print(sorted(n.name for n in rc.get_nodes()))
```

```text
b'hello from python'
['10.0.0.1:7000', '10.0.0.1:7001', '10.0.0.1:7002', '10.0.0.2:7000', '10.0.0.2:7001', '10.0.0.2:7002']
```

The client verified each node's name against its certificate, which is why
the certificates list the IP addresses. The replies are bytes unless you pass
`decode_responses=True`. Clients over TLS are told TLS ports in redirects, in
`CLUSTER SLOTS` and in `CLUSTER NODES`. Other libraries are covered in
[Clients](https://docs.hotkv.com/clustering/clients/index.md). If your applications cannot present a
certificate, `tls-auth-clients optional` or `no` relaxes the check for
clients only; between nodes it never applies (see [TLS](https://docs.hotkv.com/operate/tls/index.md)).

## 6. Reshard under TLS

`redis-cli --cluster reshard` and the other cluster tools work over TLS with
the same options. This example moves 2000 slots from one primary to another, with
200 keys in the cluster. The keys in those slots move with them:

```bash
rcli() { redis-cli --tls --cacert /etc/hotkv/tls/ca.crt --cert /etc/hotkv/tls/node.crt --key /etc/hotkv/tls/node.key "$@"; }
rcli --cluster check 10.0.0.1:7000 | head -n 4
```

```text
10.0.0.1:7000 (4b4522cf...) -> 65 keys | 5361 slots | 1 slaves.
10.0.0.2:7000 (d397b1a9...) -> 61 keys | 5462 slots | 1 slaves.
10.0.0.1:7001 (9a93f72e...) -> 75 keys | 5561 slots | 1 slaves.
[OK] 201 keys in 3 masters.
```

```bash
FROM=$(rcli -h 10.0.0.1 -p 7000 CLUSTER MYID)
TO=$(rcli -h 10.0.0.1 -p 7001 CLUSTER MYID)
rcli --cluster reshard 10.0.0.1:7000 --cluster-from $FROM --cluster-to $TO --cluster-slots 2000 --cluster-yes | tail -n 3
```

```text
Moving slot 2097 from 10.0.0.1:7000 to 10.0.0.1:7001: 
Moving slot 2098 from 10.0.0.1:7000 to 10.0.0.1:7001: 
Moving slot 2099 from 10.0.0.1:7000 to 10.0.0.1:7001: 
```

```bash
rcli --cluster check 10.0.0.1:7000 | head -n 4
```

```text
10.0.0.1:7000 (4b4522cf...) -> 46 keys | 3361 slots | 1 slaves.
10.0.0.2:7000 (d397b1a9...) -> 61 keys | 5462 slots | 1 slaves.
10.0.0.1:7001 (9a93f72e...) -> 94 keys | 7561 slots | 1 slaves.
[OK] 201 keys in 3 masters.
```

(The `rcli` function stands for the full `redis-cli` command with the TLS
options, and `REDISCLI_AUTH` was set.) Moving keys between nodes also uses the
TLS client ports when the nodes go by their TLS ports. The other reshard
methods are in [Scaling](https://docs.hotkv.com/clustering/scaling/index.md).

## 7. Renew certificates

A node loads its certificate when it starts, and again when you tell it to.
Replace the file, then reload, one node at a time, with no restart and no
downtime.

The commands below use the `rcli` function from the previous section. Look
at the certificate a node presents, issue a new one from the same key, and
replace the file:

```bash
openssl s_client -connect 10.0.0.1:7000 -CAfile /etc/hotkv/tls/ca.crt </dev/null 2>/dev/null | openssl x509 -noout -serial -enddate
```

```text
serial=1A2B3C4D5E6F708192A3B4C5D6E7F8091A2B3C4D
notAfter=Jan  1 00:00:00 2028 GMT
```

```bash
(cd pki
openssl req -new -key node1.key -subj "/CN=node1.example.com" -out node1.csr
openssl x509 -req -in node1.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -days 400 \
  -extfile <(printf "subjectAltName=DNS:node1.example.com,IP:10.0.0.1,IP:127.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\n") \
  -out node1.crt
install -m 644 node1.crt /etc/hotkv/tls/node.crt)
```

The node keeps presenting the old certificate (the same serial) until it
reloads. Setting `tls-cert-file` to the path it already has reloads it, and
`CONFIG SET` takes effect at once:

```bash
for port in 7000 7001 7002; do
  rcli -h 10.0.0.1 -p $port CONFIG SET tls-cert-file /etc/hotkv/tls/node.crt
done
```

```text
OK
OK
OK
```

```bash
openssl s_client -connect 10.0.0.1:7000 -CAfile /etc/hotkv/tls/ca.crt </dev/null 2>/dev/null | openssl x509 -noout -serial -enddate
rcli -h 10.0.0.1 -p 7000 CLUSTER INFO | head -n 2
```

```text
serial=1A2B3C4D5E6F708192A3B4C5D6E7F8091A2B3C4E
notAfter=Feb  5 00:00:00 2028 GMT
cluster_enabled:1
cluster_state:ok
```

New connections use the new certificate, and connections that are already up
keep the old one until they close. If the new file cannot be loaded, the
command fails and the node keeps the certificate it had. Here the file held
text instead of a certificate (a separate file, so that the certificate in use
stays intact):

```bash
echo "not a certificate" > /tmp/not-a-cert.crt
rcli -h 10.0.0.1 -p 7000 CONFIG SET tls-cert-file /tmp/not-a-cert.crt
```

```text
ERR Unable to update TLS configuration: use_certificate_chain_file: no start line (PEM routines) [...]
```

(The message ends with an internal error code, left out here.)

To avoid running the command after every renewal, let the node look for
changes itself with `tls-auto-reload-interval` (seconds, `0` is off, the
default). It checks the certificate, key, CA and DH files, and reloads when one
changed. A renewal job then only replaces the files:

```bash
rcli -h 10.0.0.1 -p 7000 CONFIG SET tls-auto-reload-interval 5
rcli -h 10.0.0.1 -p 7000 CONFIG GET tls-auto-reload-interval
```

```text
OK
tls-auto-reload-interval
5
```

A few seconds after a new `node.crt` is copied in, the node logs this and
presents the new serial:

```text
TLS configuration reloaded (cert=/etc/hotkv/tls/node.crt, key=/etc/hotkv/tls/node.key)
TLS materials reloaded successfully
```

A reload that fails is logged and the certificates in use stay:

```text
TLS reload failed, keeping the current certificate: use_certificate_chain_file: no start line (PEM routines) [...]
```

(The line ends with the same internal error code, left out here.)

Renew every node's certificate well before it expires.

## Bus protection

Without `tls-cluster`, any host that can reach the bus port can join the
cluster and speak its protocol. Each such node logs this at startup:

```text
WARNING: the cluster bus port is not authenticated: tls-cluster is off and cluster-bus-port-protected-mode is no, so any host that reaches the port can join this cluster and speak its protocol. Let only trusted hosts reach it, or set tls-cluster yes.
```

`cluster-bus-port-protected-mode yes` makes a node refuse to run in that
state. Without TLS on the bus it does not start:

```bash
hotkv --port 7053 --bind 127.0.0.1 --cluster-enabled yes --cluster-bus-port-protected-mode yes --dir /var/lib/hotkv/7052 2>&1 | head -n 4
```

```text

*** FATAL CONFIG FILE ERROR ***
cluster-bus-port-protected-mode is enabled but tls-cluster is disabled: the cluster bus port of this node would take any host that reaches it as a node of the cluster. Set 'tls-cluster yes', so that the other nodes must show a certificate tls-ca-cert-file trusts, or set 'cluster-bus-port-protected-mode no' if only trusted hosts can reach the port (a firewall, say)
```

With `tls-cluster yes` it starts without the warning, and turning TLS off at
runtime is refused:

```text
$ redis-cli --tls ... -p 7001 CONFIG GET cluster-bus-port-protected-mode
cluster-bus-port-protected-mode
yes
$ redis-cli --tls ... -p 7001 CONFIG SET tls-cluster no
ERR CONFIG SET failed (possibly related to argument 'tls-cluster') - can't disable tls-cluster while cluster-bus-port-protected-mode is enabled: the bus would take any host; set cluster-bus-port-protected-mode no first, or in the same command
```

(The two commands carry the same `--cacert`, `--cert` and `--key` options as
before, and `REDISCLI_AUTH`.) `tls-expected-peer-name` adds a name check: a
list of names, one of which the other node's certificate must carry (as a DNS
name, or as its common name when it has none). It applies to the links this
node opens and to the bus connections it accepts. A mismatch fails like this
(see the next section). The setting takes effect from the next connection and
changes at runtime:

```bash
rcli -h 10.0.0.1 -p 7000 CONFIG SET tls-expected-peer-name other.example.com
```

## When the handshake fails

A refused link is logged on both ends, with the reason. The node that opens
the link logs `the TLS bus link to node ... failed`, and the node that
accepts it logs `Error accepting cluster node connection`. Both lines end
with a reminder of what a bus link needs. Each case below happens when you
introduce a new node with `CLUSTER MEET`.

**A certificate from another CA.** The node that opened the link:

```text
Cluster: the TLS bus link to node bb6d9054e9709ce7ab565cc44430652ee6d574a3 at 10.0.0.1:17050 failed: certificate verify failed (SSL routines): unable to get local issuer certificate (the TLS handshake: it must have tls-cluster yes, and a certificate tls-ca-cert-file trusts)
```

The node that was met:

```text
Error accepting cluster node connection: tlsv1 alert unknown ca (SSL routines) (the TLS handshake from 10.0.0.1: tls-cluster is on here, the other nodes must use it too, with a certificate tls-ca-cert-file trusts)
```

**An expired certificate.**

```text
Cluster: the TLS bus link to node 083ebe900a8561def80a0f6b09ddc855fafdaf4c at 10.0.0.1:17051 failed: certificate verify failed (SSL routines): certificate has expired (the TLS handshake: it must have tls-cluster yes, and a certificate tls-ca-cert-file trusts)
Error accepting cluster node connection: ssl/tls alert certificate expired (SSL routines) (the TLS handshake from 10.0.0.1: tls-cluster is on here, the other nodes must use it too, with a certificate tls-ca-cert-file trusts)
```

**A name that does not match `tls-expected-peer-name`.**

```text
Cluster: the TLS bus link to node 1792999059702fbffadf2f0ba066ed9246d2ca3a at 10.0.0.1:17052 failed: certificate verify failed (SSL routines): hostname mismatch (the TLS handshake: it must have tls-cluster yes, and a certificate tls-ca-cert-file trusts)
```

**A node without `tls-cluster`.** The TLS nodes log the `wrong version number`
line from section 4 for it. A plain node that is told about TLS nodes does not
get an answer it can read, and logs this for each of them:

```text
Cluster link message too large from 964febbb1fe2c12407d497160f6ead5514e840f4: 352518912 bytes
```

In all cases the new node does not join the cluster. **Fix.** Issue the node a certificate that your CA signed and that has not
expired, set `tls-cluster yes` (and `tls-replication yes`) on it, and check
the names if you use `tls-expected-peer-name`. A node that does not start
because of its certificate or key files names the file in its log. The other
errors in [Troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md) apply here too.

> **Note**
> A TLS client that cannot verify a node's certificate fails before any
> cluster command runs. `redis-cli` prints `Could not connect to Redis at
> 10.0.0.1:7050: SSL_connect failed: certificate verify failed` when the node
> presents a certificate from another CA.

## Next steps

- [TLS](https://docs.hotkv.com/operate/tls/index.md): TLS for a single server, protocols and ciphers, and
  logging in by certificate.
- [Production checklist](https://docs.hotkv.com/clustering/production-checklist/index.md): where TLS fits in
  the list of things to check.
- [Cluster configuration](https://docs.hotkv.com/clustering/configuration/index.md): every TLS and cluster
  setting.