# Cluster production checklist > A checklist to go through before a HotKV cluster takes production traffic, with each item linked to the page that explains and verifies it. URL: https://docs.hotkv.com/clustering/production-checklist/ **Availability:** Not on HotKV Cloud. On-Premise: no license needed. Go through this list before you send production traffic to a cluster, and again after any big change. Each item says what to check and links to the page that explains it and shows the commands. ## Sizing and layout - [ ] At least 3 primaries. `redis-cli --cluster create` refuses fewer, and a cluster with two primaries stops when either fails. [Overview](https://docs.hotkv.com/clustering/index.md#how-big-should-it-be) - [ ] At least one replica per primary. [Plan the layout](https://docs.hotkv.com/clustering/production/index.md#plan-the-layout) - [ ] Each primary and its replicas are on different hosts, and in different availability zones when you have them. At least three hosts, so that losing one host leaves a majority of the primaries. [Plan the layout](https://docs.hotkv.com/clustering/production/index.md#plan-the-layout) - [ ] `maxmemory` is set below the machine's memory, with room for replication buffers. [Plan the layout](https://docs.hotkv.com/clustering/production/index.md#plan-the-layout) - [ ] You know how you will add primaries when the data grows. [Scaling](https://docs.hotkv.com/clustering/scaling/index.md) ## Hosts and operating system - [ ] The open files limit is high enough for `maxclients` (the log says when it is not): `LimitNOFILE` in the unit, or `ulimit -n`. [Open files](https://docs.hotkv.com/clustering/production/index.md#open-files) - [ ] Clocks are synchronised with NTP or chrony on every node. [Clocks](https://docs.hotkv.com/clustering/production/index.md#clocks) - [ ] Each node runs under a service manager that restarts it and has its own data directory. [Run nodes under systemd](https://docs.hotkv.com/clustering/production/index.md#run-nodes-under-systemd) - [ ] Every node has its own `dir`, `cluster-config-file` and log file. [Configure a node](https://docs.hotkv.com/clustering/production/index.md#configure-a-node) ## Network and firewall - [ ] The client port is open to your application and admin hosts and to the other nodes. [Ports and firewalls](https://docs.hotkv.com/clustering/production/index.md#ports-and-firewalls) - [ ] The bus port (client port + 10000, or the TLS port + 10000) is open between the nodes only, in both directions, and not to the internet. [Ports and firewalls](https://docs.hotkv.com/clustering/production/index.md#ports-and-firewalls) - [ ] From each host, `redis-cli -h -p PING` answers quickly instead of timing out. [Create waits forever](https://docs.hotkv.com/clustering/troubleshooting/index.md#create-waits-forever) - [ ] Nodes announce addresses that your clients can reach: names, a NAT or a container network are covered by the announce settings. [Announce addresses and NAT](https://docs.hotkv.com/clustering/production/index.md#announce-addresses-and-nat) ## Authentication and protected mode - [ ] The same `requirepass` and `masterauth` are on every node (`masterauth` is what atomic slot migration signs in with, and an ACL file does not replace either), and the clients that use the cluster have their own ACL users, created on every node. [Authentication and protected mode](https://docs.hotkv.com/clustering/production/index.md#authentication-and-protected-mode) - [ ] `protected-mode` is on (the default) unless the network already keeps everyone else out. [DENIED, protected mode](https://docs.hotkv.com/clustering/troubleshooting/index.md#denied-protected-mode) - [ ] Passwords are given to tools through `REDISCLI_AUTH`, not `-a`. [Authentication and protected mode](https://docs.hotkv.com/clustering/production/index.md#authentication-and-protected-mode) ## TLS - [ ] If any traffic crosses a network you do not control, `tls-cluster` and `tls-replication` are on for every node, and clients use TLS. [Cluster over TLS](https://docs.hotkv.com/clustering/tls/index.md) - [ ] If the bus is plain, only trusted hosts can reach it, and you accept the startup warning. To make the node refuse a plain bus, set `cluster-bus-port-protected-mode yes`. [Bus protection](https://docs.hotkv.com/clustering/tls/index.md#bus-protection) - [ ] You know when each certificate expires, and renewal is automated and tested: replace the file, then reload with `CONFIG SET` or `tls-auto-reload-interval`. [Renew certificates](https://docs.hotkv.com/clustering/tls/index.md#7-renew-certificates) ## Persistence and backups - [ ] `appendonly yes` (or snapshots at the interval you can afford to lose) on every node, and you know what `appendfsync` costs and saves. [Persistence settings first](https://docs.hotkv.com/clustering/backup-and-restore/index.md#persistence-settings-first) - [ ] A backup of every node's `dump.hkv` and `nodes.conf` runs on a schedule and is stored off the host. [Take a backup](https://docs.hotkv.com/clustering/backup-and-restore/index.md#take-a-backup) - [ ] The license files and encryption keys of each node are backed up by your configuration management. [What a node's state is](https://docs.hotkv.com/clustering/backup-and-restore/index.md#what-a-nodes-state-is) - [ ] You have restored a node, a shard and the whole cluster from a backup at least once, on a test cluster. [Backup and restore](https://docs.hotkv.com/clustering/backup-and-restore/index.md) ## Monitoring - [ ] You alert on `cluster_state` not `ok`, nodes flagged `fail`, and replicas with `master_link_status` not `up`. [Monitoring](https://docs.hotkv.com/clustering/monitoring/index.md) - [ ] You collect the node logs, and search for the lines that show failures and failovers. [Where the logs are](https://docs.hotkv.com/clustering/troubleshooting/index.md#where-the-logs-are-and-what-to-look-for) ## Failover drills - [ ] You have stopped a primary on purpose, watched a replica take over, and measured how long writes failed. Your timeout (`cluster-node-timeout`) fits what the application can wait. [Failure detection](https://docs.hotkv.com/clustering/production/index.md#failure-detection) - [ ] You decided `cluster-require-full-coverage`: stop everything when a shard is lost (`yes`), or keep serving the rest (`no`). [Failure detection](https://docs.hotkv.com/clustering/production/index.md#failure-detection) - [ ] You know what a primary can lose when it dies, and use `WAIT` or `WAITAOF` for the writes that must not be lost. [Replicas and failover](https://docs.hotkv.com/clustering/replicas-and-failover/index.md) - [ ] You have run `redis-cli --cluster check` against the cluster and it is clean. [Nodes disagree](https://docs.hotkv.com/clustering/troubleshooting/index.md#nodes-disagree) ## Upgrades - [ ] You have a rolling upgrade plan and have rehearsed it on a test cluster. [Upgrades](https://docs.hotkv.com/clustering/upgrades/index.md) ## Clients - [ ] Applications use a cluster-aware client that follows `MOVED` and `ASK`, refreshes its slot map and retries through a failover. [Clients](https://docs.hotkv.com/clustering/clients/index.md) - [ ] Multi-key commands, transactions and scripts use hash tags so that their keys share a slot. [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md) - [ ] The client is given more than one seed node, and the right credentials and TLS settings. [Clients](https://docs.hotkv.com/clustering/clients/index.md) - [ ] Anything that lists or counts keys (`KEYS`, `SCAN`, `DBSIZE`) knows that it answers for one node. [Keys and commands](https://docs.hotkv.com/clustering/keys-and-commands/index.md)