Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Troubleshooting

Last updated

Start with the error text: HotKV uses the same error prefixes as Redis (ERR, WRONGTYPE, OOM, LOADING, MOVED and so on), and most messages say what to change. This page lists the ones operators run into most, then covers latency problems and where to find the logs. The full list of error replies is in Errors.

Connection and authentication

Section titled “Connection and authentication”

DENIED: protected mode

Section titled “DENIED: protected mode”
(error) DENIED HotKV is running in protected mode because protected mode is on and the default user has no password: only clients on the loopback interface are served. ...

Protected mode is on (the default) and the server has no password, so it serves only clients on 127.0.0.1 or ::1, whatever bind says. The connection is closed after the error. Fix it by either:

  • setting a password, which is the recommended fix: requirepass in the configuration file, or CONFIG SET requirepass ... from the server’s own host; or
  • turning protected mode off where the network already keeps strangers out: protected-mode no in the file, --protected-mode no on the command line, or CONFIG SET protected-mode no from the loopback interface (then CONFIG REWRITE to keep it).

The same rule applies to replicas connecting to a primary’s client port. See Security and DENIED, protected mode.

NOAUTH and WRONGPASS

Section titled “NOAUTH and WRONGPASS”
  • NOAUTH Authentication required.: the server has a password and the client sent a command before authenticating. Pass the password in your client’s connection options, or -a (and --user) to redis-cli.
  • WRONGPASS invalid username-password pair or user is disabled.: the password, the user name, or the user’s on/off state is wrong. Check the user with ACL GETUSER name. See ACL.

ERR max number of clients reached

Section titled “ERR max number of clients reached”

The server already has maxclients connections (default 10000). New connections get this error and are closed, and rejected_connections in INFO stats goes up. Look for clients that open connections without closing them (CLIENT LIST, sorted by age or idle), use connection pooling, set timeout to close idle clients, or raise maxclients.

HotKV raises the process’s open files limit to fit maxclients when it can. When the operating system does not allow that, it lowers maxclients at startup and logs The open files limit only allows N clients. Raise the limit with ulimit -n or LimitNOFILE= in the systemd unit (the packaged unit sets 65535).

The server does not start

Section titled “The server does not start”

Check the log (see Logs). The common causes:

  • FATAL CONFIG FILE ERROR: a line in hotkv.conf names an unknown directive, or a directive HotKV refuses (replicaof, rename-command, user, loadmodule, sentinel). The message names the file and line. See Configuration.
  • A port is already in use by another process: the client port (Failed to bind to ... - Address already in use), the TLS port, the cluster bus port (Failed listening on port ... (the cluster bus): Address already in use), or the replication port if you set replication.dedicated-port (Failed listening on port ... (replication)). Stop the other process or change the port. A cluster node that does not start for another reason (a damaged nodes.conf, for example) is covered in Cluster troubleshooting.
  • On the command line, unknown setting --name or --name does not take 'value': the command line refuses what the file would only warn about.

While it loads a large snapshot or AOF at startup, the server does not yet accept connections; the packaged systemd unit reports the service as started only once loading is done.

Licensing

Section titled “Licensing”

NOLICENSE

Section titled “NOLICENSE”
(error) NOLICENSE semantic-cache requires a HotKV license. Contact sales@hotkv.com

The command belongs to a feature your license does not include, or the server has no usable license. The core data store keeps working; only licensed features are refused. Check the license state and features with LICENSE.INFO, and see Quick start: On-Premise. On HotKV Cloud, the features follow your plan: upgrade the plan to unlock them.

Cluster errors

Section titled “Cluster errors”

These replies come from a node in cluster mode. This is the short form; Cluster troubleshooting has the exact errors and log lines, the causes, the fixes and how to confirm them.

MOVED and ASK

Section titled “MOVED and ASK”
(error) MOVED 3999 10.0.0.2:6379

These are not failures: the key lives on another node and the client should retry there. A client that shows them to you is not running in cluster mode. Use redis-cli -c, or your library’s cluster client (every HotKV SDK has one); they follow MOVED and ASK automatically. See Clustering. If the address in MOVED cannot be reached from where the client runs, see Redirected to an unreachable address.

CROSSSLOT

Section titled “CROSSSLOT”

CROSSSLOT Keys in request don't hash to the same slot: a multi-key command, transaction or script used keys from different hash slots. Put related keys in the same slot with a hash tag, for example {user:42}:cart and {user:42}:profile.

CLUSTERDOWN

Section titled “CLUSTERDOWN”
Message Cause and fix
CLUSTERDOWN The cluster is down Some slots have no serving primary while cluster-require-full-coverage is yes, or this node is on the minority side of a partition. Check CLUSTER INFO (cluster_state, cluster_slots_ok) and CLUSTER NODES for failed nodes. It clears once a replica takes over or the node rejoins the majority.
CLUSTERDOWN Hash slot not served No node owns this key’s slot. redis-cli --cluster check lists the uncovered slots; see Nodes disagree.
CLUSTERDOWN The cluster is down and only accepts read commands The cluster is down and cluster-allow-reads-when-down is yes: reads work, writes wait until it recovers.

TRYAGAIN Multiple keys request during rehashing of slot means a multi-key command hit a slot in the middle of a migration; retry after a short delay. See High availability.

Data errors

Section titled “Data errors”

OOM

Section titled “OOM”
(error) OOM command not allowed when used memory > 'maxmemory'.

Memory is at maxmemory and nothing can be evicted: the policy is noeviction, or a volatile-* policy finds no keys with a TTL. Reads and deletes still work. Free memory (delete or expire keys), raise maxmemory, or choose an evicting policy. See Memory and eviction.

LOADING

Section titled “LOADING”
(error) LOADING HotKV is loading the dataset in memory

On a replica, this means a full sync from its primary is replacing its data. It clears when the sync completes. PING, INFO, ROLE and similar commands keep answering meanwhile; INFO replication shows master_sync_in_progress:1. If it never clears, the sync keeps failing: check the replica’s and the primary’s logs, and the primary’s replication.replica-buffer-limit. See Replication.

MISCONF

Section titled “MISCONF”

MISCONF The last background save failed, ... or MISCONF The AOF cannot be written to disk (...): the server cannot persist, so it refuses writes rather than accept data it would lose. Usual causes are a full disk or a dir the server cannot write. The log has the reason. Writes are accepted again once a save succeeds or the AOF can be written. For snapshot failures, stop-writes-on-bgsave-error no keeps accepting writes anyway. See Persistence.

READONLY

Section titled “READONLY”

READONLY You can't write against a read only replica.: the client is connected to a replica. Send writes to the primary, or see Replication.

WRONGTYPE

Section titled “WRONGTYPE”

WRONGTYPE Operation against a key holding the wrong kind of value: for example LPUSH on a key that holds a string. Check the key with TYPE key. This usually means two parts of an application use the same key name for different data; give them distinct prefixes.

BUSY

Section titled “BUSY”

BUSY HotKV is busy running a script. ...: a Lua script or function has run longer than busy-reply-threshold (default 5000 ms). Wait for it, stop it with SCRIPT KILL (or FUNCTION KILL), or as a last resort SHUTDOWN NOSAVE.

Diagnosing latency

Section titled “Diagnosing latency”

When commands are slower than expected, work from the server outward:

  1. Is the server slow, or the network? Run redis-cli --latency from the client’s host and compare it with the same run on the server’s own host. A large difference points to the network.
  2. Which commands are slow? SLOWLOG GET 20 lists the slowest recent commands with their arguments. Look for O(N) commands on large keys (KEYS, SMEMBERS, HGETALL, LRANGE 0 -1, SORT) and replace them with SCAN-family iteration or smaller ranges.
  3. Where is time going overall? INFO commandstats gives average microseconds per call per command, and LATENCY HISTOGRAM the distribution. Set latency-monitor-threshold (for example 50 ms) and read LATENCY LATEST and LATENCY DOCTOR to see if spikes come from commands or from expiring keys (expire-cycle).
  4. Is a big key involved? MEMORY USAGE key shows a key’s size; a single very large value or collection makes every command on it slow. With hotkey-tracking yes, HOTKEYS TOP shows keys that take most of the traffic.
  5. Is memory pressure involved? total_eviction_exceeded_time and evicted_keys growing in INFO stats mean commands are evicting keys before they run. MEMORY DOCTOR summarises other memory problems.
  6. Are clients reading their replies? A large omem in CLIENT LIST means a client is slow to read its replies, and the server holds them in memory meanwhile. Normal clients have no output limit by default; set client-output-buffer-limit normal ... to disconnect such clients (Pub/Sub clients have a limit by default). maxmemory-clients closes the largest clients when they hold too much together; evicted_clients in INFO stats counts them, and the log names each one.
  7. Is persistence involved? appendfsync always waits for the disk on every write. everysec (the default) is the usual trade-off.

See Monitoring for each of these tools.

DEBUG

Section titled “DEBUG”

DEBUG is disabled by default, as in Redis 7:

(error) ERR DEBUG command not allowed. If the enable-debug-command option is set to "local", you can run it from a local connection, otherwise you need to set this option in the configuration file, and then restart the server.

enable-debug-command takes no (the default), yes, or local (only from a loopback connection). It is read at startup only. Enable it on test systems rather than in production: subcommands such as DEBUG SLEEP stop the whole server, and DEBUG RELOAD and DEBUG RESTART reload or restart it. DEBUG OBJECT key (low-level details of a key) and DEBUG HELP are the ones most useful for diagnosis.

Logs

Section titled “Logs”
Setting Default Meaning
logfile "" Log file. Empty logs to standard output; a relative path is placed inside dir.
loglevel notice debug, verbose, notice or warning

With the packaged systemd unit and the default empty logfile, the log goes to the journal:

Terminal window
journalctl -u hotkv -f

To log to a file instead, set an absolute path such as logfile /var/log/hotkv/hotkv.log; the package’s logrotate rule rotates files in /var/log/hotkv weekly. logfile is read at startup only.

debug logs the most detail; verbose logs the same as notice; warning logs only warnings and errors. Set loglevel in the configuration file and restart to change it: the level of a running server is fixed at startup, and CONFIG SET loglevel does not change it.

Next steps

Section titled “Next steps”