# Troubleshooting

> Fix common HotKV errors (DENIED, NOLICENSE, CLUSTERDOWN, MOVED, OOM, LOADING, MISCONF, max clients), diagnose latency, and find the logs.

URL: https://docs.hotkv.com/operate/troubleshooting/


Start with the error text: HotKV uses the same error prefixes as Redis
(`ERR`, `WRONGTYPE`, `OOM`, `LOADING`, `MOVED` and so on), and most messages
say what to change. This page lists the ones operators run into most, then
covers latency problems and where to find the logs. The full list of error
replies is in [Errors](https://docs.hotkv.com/reference/errors/index.md).

## Connection and authentication

### DENIED: protected mode

```
(error) DENIED HotKV is running in protected mode because protected mode is on and the default user has no password: only clients on the loopback interface are served. ...
```

Protected mode is on (the default) and the server has no password, so it
serves only clients on `127.0.0.1` or `::1`, whatever `bind` says. The
connection is closed after the error. Fix it by either:

- setting a password, which is the recommended fix: `requirepass` in the
  configuration file, or `CONFIG SET requirepass ...` from the server's
  own host; or
- turning protected mode off where the network already keeps strangers out:
  `protected-mode no` in the file, `--protected-mode no` on the command
  line, or `CONFIG SET protected-mode no` from the loopback interface
  (then `CONFIG REWRITE` to keep it).

The same rule applies to replicas connecting to a primary's client port.
See [Security](https://docs.hotkv.com/security/index.md#protected-mode) and
[DENIED, protected mode](https://docs.hotkv.com/clustering/troubleshooting/index.md#denied-protected-mode).

### NOAUTH and WRONGPASS

- `NOAUTH Authentication required.`: the server has a password and the
  client sent a command before authenticating. Pass the password in your
  client's connection options, or `-a` (and `--user`) to `redis-cli`.
- `WRONGPASS invalid username-password pair or user is disabled.`: the
  password, the user name, or the user's `on`/`off` state is wrong. Check
  the user with `ACL GETUSER name`. See [ACL](https://docs.hotkv.com/operate/acl/index.md).

### ERR max number of clients reached

The server already has `maxclients` connections (default 10000). New
connections get this error and are closed, and `rejected_connections` in
`INFO stats` goes up. Look for clients that open connections without
closing them (`CLIENT LIST`, sorted by `age` or `idle`), use connection
pooling, set `timeout` to close idle clients, or raise `maxclients`.

HotKV raises the process's open files limit to fit `maxclients` when it can.
When the operating system does not allow that, it lowers `maxclients` at
startup and logs `The open files limit only allows N clients`. Raise the
limit with `ulimit -n` or `LimitNOFILE=` in the systemd unit (the packaged
unit sets 65535).

### The server does not start

Check the log (see [Logs](#logs)). The common causes:

- `FATAL CONFIG FILE ERROR`: a line in `hotkv.conf` names an unknown
  directive, or a directive HotKV refuses (`replicaof`, `rename-command`,
  `user`, `loadmodule`, `sentinel`). The message names the file and line.
  See [Configuration](https://docs.hotkv.com/operate/configuration/index.md#how-mistakes-are-handled).
- A port is already in use by another process: the client port (`Failed to
  bind to ... - Address already in use`), the TLS port, the cluster bus port
  (`Failed listening on port ... (the cluster bus): Address already in use`),
  or the replication port if you set `replication.dedicated-port` (`Failed
  listening on port ... (replication)`). Stop the other process or change the
  port. A cluster node that does not start for another reason (a damaged
  `nodes.conf`, for example) is covered in
  [Cluster troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md#a-node-does-not-start).
- On the command line, `unknown setting --name` or `--name does not take
  'value'`: the command line refuses what the file would only warn about.

While it loads a large snapshot or AOF at startup, the server does not yet
accept connections; the packaged systemd unit reports the service as started
only once loading is done.

## Licensing

### NOLICENSE

```
(error) NOLICENSE semantic-cache requires a HotKV license. Contact sales@hotkv.com
```

The command belongs to a feature your license does not include, or the
server has no usable license. The core data store keeps working; only
licensed features are refused. Check the license state and features with
`LICENSE.INFO`, and see
[Quick start: On-Premise](https://docs.hotkv.com/quickstart-onpremise/index.md#5-check-your-license-status).
On HotKV Cloud, the features follow your plan: upgrade the plan to unlock
them.

## Cluster errors

These replies come from a node in [cluster mode](https://docs.hotkv.com/clustering/index.md). This is the
short form; [Cluster troubleshooting](https://docs.hotkv.com/clustering/troubleshooting/index.md) has the
exact errors and log lines, the causes, the fixes and how to confirm them.

### MOVED and ASK

```
(error) MOVED 3999 10.0.0.2:6379
```

These are not failures: the key lives on another node and the client should
retry there. A client that shows them to you is not running in cluster
mode. Use `redis-cli -c`, or your library's cluster client (every HotKV SDK
has one); they follow `MOVED` and `ASK` automatically. See
[Clustering](https://docs.hotkv.com/clustering/index.md#slots-and-redirects). If the address in `MOVED`
cannot be reached from where the client runs, see
[Redirected to an unreachable address](https://docs.hotkv.com/clustering/troubleshooting/index.md#redirected-to-an-unreachable-address).

### CROSSSLOT

`CROSSSLOT Keys in request don't hash to the same slot`: a multi-key
command, transaction or script used keys from different hash slots. Put
related keys in the same slot with a hash tag, for example `{user:42}:cart`
and `{user:42}:profile`.

### CLUSTERDOWN

| Message | Cause and fix |
| --- | --- |
| `CLUSTERDOWN The cluster is down` | Some slots have no serving primary while `cluster-require-full-coverage` is `yes`, or this node is on the minority side of a partition. Check `CLUSTER INFO` (`cluster_state`, `cluster_slots_ok`) and `CLUSTER NODES` for failed nodes. It clears once a replica takes over or the node rejoins the majority. |
| `CLUSTERDOWN Hash slot not served` | No node owns this key's slot. `redis-cli --cluster check` lists the uncovered slots; see [Nodes disagree](https://docs.hotkv.com/clustering/troubleshooting/index.md#nodes-disagree). |
| `CLUSTERDOWN The cluster is down and only accepts read commands` | The cluster is down and `cluster-allow-reads-when-down` is `yes`: reads work, writes wait until it recovers. |

`TRYAGAIN Multiple keys request during rehashing of slot` means a multi-key
command hit a slot in the middle of a migration; retry after a short delay.
See [High availability](https://docs.hotkv.com/operate/high-availability/index.md).

## Data errors

### OOM

```
(error) OOM command not allowed when used memory > 'maxmemory'.
```

Memory is at `maxmemory` and nothing can be evicted: the policy is
`noeviction`, or a `volatile-*` policy finds no keys with a TTL. Reads and
deletes still work. Free memory (delete or expire keys), raise `maxmemory`,
or choose an evicting policy. See
[Memory and eviction](https://docs.hotkv.com/operate/memory-and-eviction/index.md).

### LOADING

```
(error) LOADING HotKV is loading the dataset in memory
```

On a replica, this means a full sync from its primary is replacing its data.
It clears when the sync completes. `PING`, `INFO`, `ROLE` and similar
commands keep answering meanwhile; `INFO replication` shows
`master_sync_in_progress:1`. If it never clears, the sync keeps failing:
check the replica's and the primary's logs, and the primary's
`replication.replica-buffer-limit`. See
[Replication](https://docs.hotkv.com/operate/replication/index.md#full-sync-and-partial-resync).

### MISCONF

`MISCONF The last background save failed, ...` or `MISCONF The AOF cannot be
written to disk (...)`: the server cannot persist, so it refuses writes
rather than accept data it would lose. Usual causes are a full disk or a
`dir` the server cannot write. The log has the reason. Writes are accepted
again once a save succeeds or the AOF can be written. For snapshot
failures, `stop-writes-on-bgsave-error no` keeps accepting writes anyway.
See [Persistence](https://docs.hotkv.com/persistence/index.md).

### READONLY

`READONLY You can't write against a read only replica.`: the client is
connected to a replica. Send writes to the primary, or see
[Replication](https://docs.hotkv.com/operate/replication/index.md#read-only-replicas).

### WRONGTYPE

`WRONGTYPE Operation against a key holding the wrong kind of value`: for
example `LPUSH` on a key that holds a string. Check the key with `TYPE key`.
This usually means two parts of an application use the same key name for
different data; give them distinct prefixes.

### BUSY

`BUSY HotKV is busy running a script. ...`: a Lua script or function has run
longer than `busy-reply-threshold` (default 5000 ms). Wait for it, stop it
with `SCRIPT KILL` (or `FUNCTION KILL`), or as a last resort
`SHUTDOWN NOSAVE`.

## Diagnosing latency

When commands are slower than expected, work from the server outward:

1. **Is the server slow, or the network?** Run `redis-cli --latency` from
   the client's host and compare it with the same run on the server's own
   host. A large difference points to the network.
2. **Which commands are slow?** `SLOWLOG GET 20` lists the slowest recent
   commands with their arguments. Look for O(N) commands on large keys
   (`KEYS`, `SMEMBERS`, `HGETALL`, `LRANGE 0 -1`, `SORT`) and replace them
   with `SCAN`-family iteration or smaller ranges.
3. **Where is time going overall?** `INFO commandstats` gives average
   microseconds per call per command, and `LATENCY HISTOGRAM` the
   distribution. Set `latency-monitor-threshold` (for example 50 ms) and
   read `LATENCY LATEST` and `LATENCY DOCTOR` to see if spikes come from
   commands or from expiring keys (`expire-cycle`).
4. **Is a big key involved?** `MEMORY USAGE key` shows a key's size; a
   single very large value or collection makes every command on it slow.
   With `hotkey-tracking yes`, `HOTKEYS TOP` shows keys that take most of
   the traffic.
5. **Is memory pressure involved?** `total_eviction_exceeded_time` and
   `evicted_keys` growing in `INFO stats` mean commands are evicting keys
   before they run. `MEMORY DOCTOR` summarises other memory problems.
6. **Are clients reading their replies?** A large `omem` in `CLIENT LIST`
   means a client is slow to read its replies, and the server holds them in
   memory meanwhile. Normal clients have no output limit by default; set
   `client-output-buffer-limit normal ...` to disconnect such clients
   (Pub/Sub clients have a limit by default). `maxmemory-clients` closes the
   largest clients when they hold too much together; `evicted_clients` in
   `INFO stats` counts them, and the log names each one.
7. **Is persistence involved?** `appendfsync always` waits for the disk on
   every write. `everysec` (the default) is the usual trade-off.

See [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md) for each of these tools.

## DEBUG

[`DEBUG`](https://docs.hotkv.com/commands/debug/index.md) is disabled by default, as in Redis 7:

```
(error) ERR DEBUG command not allowed. If the enable-debug-command option is set to "local", you can run it from a local connection, otherwise you need to set this option in the configuration file, and then restart the server.
```

`enable-debug-command` takes `no` (the default), `yes`, or `local` (only
from a loopback connection). It is read at startup only. Enable it on test
systems rather than in production: subcommands such as `DEBUG SLEEP` stop
the whole server, and `DEBUG RELOAD` and `DEBUG RESTART` reload or restart
it. `DEBUG OBJECT key` (low-level details of a key) and `DEBUG HELP` are
the ones most useful for diagnosis.

## Logs

| Setting | Default | Meaning |
| --- | --- | --- |
| `logfile` | `""` | Log file. Empty logs to standard output; a relative path is placed inside `dir`. |
| `loglevel` | `notice` | `debug`, `verbose`, `notice` or `warning` |

With the packaged systemd unit and the default empty `logfile`, the log goes
to the journal:

```bash
journalctl -u hotkv -f
```

To log to a file instead, set an absolute path such as
`logfile /var/log/hotkv/hotkv.log`; the package's logrotate rule rotates
files in `/var/log/hotkv` weekly. `logfile` is read at startup only.

`debug` logs the most detail; `verbose` logs the same as `notice`; `warning` logs only warnings and errors. Set `loglevel` in the
configuration file and restart to change it: the level of a running server
is fixed at startup, and `CONFIG SET loglevel` does not change it.

> **Tip**
> When you open a support ticket, include the HotKV version (`hotkv
> --version` or `INFO server`), the relevant log lines, and the output of
> `INFO everything` if you can share it.

## Next steps

- [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md)
- [Errors reference](https://docs.hotkv.com/reference/errors/index.md)
- [Configuration](https://docs.hotkv.com/operate/configuration/index.md)