Cluster production checklist

Not on HotKV CloudOn-Premise: no license needed

Go through this list before you send production traffic to a cluster, and again after any big change. Each item says what to check and links to the page that explains it and shows the commands.

Sizing and layout

• At least 3 primaries. redis-cli --cluster create refuses fewer, and a cluster with two primaries stops when either fails. Overview
• At least one replica per primary. Plan the layout
• Each primary and its replicas are on different hosts, and in different availability zones when you have them. At least three hosts, so that losing one host leaves a majority of the primaries. Plan the layout
• maxmemory is set below the machine’s memory, with room for replication buffers. Plan the layout
• You know how you will add primaries when the data grows. Scaling

Hosts and operating system

• The open files limit is high enough for maxclients (the log says when it is not): LimitNOFILE in the unit, or ulimit -n. Open files
• Clocks are synchronised with NTP or chrony on every node. Clocks
• Each node runs under a service manager that restarts it and has its own data directory. Run nodes under systemd
• Every node has its own dir, cluster-config-file and log file. Configure a node

Network and firewall

• The client port is open to your application and admin hosts and to the other nodes. Ports and firewalls
• The bus port (client port + 10000, or the TLS port + 10000) is open between the nodes only, in both directions, and not to the internet. Ports and firewalls
• From each host, redis-cli -h <other host> -p <bus port> PING answers quickly instead of timing out. Create waits forever
• Nodes announce addresses that your clients can reach: names, a NAT or a container network are covered by the announce settings. Announce addresses and NAT

Authentication and protected mode

• The same requirepass and masterauth are on every node (masterauth is what atomic slot migration signs in with, and an ACL file does not replace either), and the clients that use the cluster have their own ACL users, created on every node. Authentication and protected mode
• protected-mode is on (the default) unless the network already keeps everyone else out. DENIED, protected mode
• Passwords are given to tools through REDISCLI_AUTH, not -a. Authentication and protected mode

TLS

• If any traffic crosses a network you do not control, tls-cluster and tls-replication are on for every node, and clients use TLS. Cluster over TLS
• If the bus is plain, only trusted hosts can reach it, and you accept the startup warning. To make the node refuse a plain bus, set cluster-bus-port-protected-mode yes. Bus protection
• You know when each certificate expires, and renewal is automated and tested: replace the file, then reload with CONFIG SET or tls-auto-reload-interval. Renew certificates

Persistence and backups

• appendonly yes (or snapshots at the interval you can afford to lose) on every node, and you know what appendfsync costs and saves. Persistence settings first
• A backup of every node’s dump.hkv and nodes.conf runs on a schedule and is stored off the host. Take a backup
• The license files and encryption keys of each node are backed up by your configuration management. What a node’s state is
• You have restored a node, a shard and the whole cluster from a backup at least once, on a test cluster. Backup and restore

Monitoring

• You alert on cluster_state not ok, nodes flagged fail, and replicas with master_link_status not up. Monitoring
• You collect the node logs, and search for the lines that show failures and failovers. Where the logs are

Failover drills

• You have stopped a primary on purpose, watched a replica take over, and measured how long writes failed. Your timeout (cluster-node-timeout) fits what the application can wait. Failure detection
• You decided cluster-require-full-coverage: stop everything when a shard is lost (yes), or keep serving the rest (no). Failure detection
• You know what a primary can lose when it dies, and use WAIT or WAITAOF for the writes that must not be lost. Replicas and failover
• You have run redis-cli --cluster check against the cluster and it is clean. Nodes disagree

Upgrades

• You have a rolling upgrade plan and have rehearsed it on a test cluster. Upgrades

Clients

• Applications use a cluster-aware client that follows MOVED and ASK, refreshes its slot map and retries through a failover. Clients
• Multi-key commands, transactions and scripts use hash tags so that their keys share a slot. Keys and commands
• The client is given more than one seed node, and the right credentials and TLS settings. Clients
• Anything that lists or counts keys (KEYS, SCAN, DBSIZE) knows that it answers for one node. Keys and commands