# Monitoring

> Watch HotKV with INFO, the Prometheus metrics endpoint, SLOWLOG, LATENCY, CLIENT LIST, hot key tracking and the enterprise OBSERVE and ALERTS commands.

URL: https://docs.hotkv.com/operate/monitoring/


HotKV exposes the same monitoring surface as Redis (`INFO`, `SLOWLOG`,
`LATENCY`, `CLIENT LIST`, `MONITOR`), plus a Prometheus endpoint, hot key
tracking, and enterprise observability commands. This page covers what each
one tells you and how to turn it on.

## INFO

[`INFO`](https://docs.hotkv.com/commands/info/index.md) returns the server's state as text, in sections.
It keeps Redis's section names, field names and order, so tools written
against Redis's `INFO` find the fields where they expect them.

```
127.0.0.1:6379> INFO memory
```

| Section | What it covers |
| --- | --- |
| `server` | Version (`redis_version` for tool compatibility, and `hotkv_version`), process, uptime, ports, config file, worker threads |
| `clients` | Connected and blocked clients, `maxclients`, largest input and output buffers |
| `memory` | `used_memory`, RSS, peak, `maxmemory`, policy, fragmentation; see [Memory and eviction](https://docs.hotkv.com/operate/memory-and-eviction/index.md) |
| `persistence` | Snapshot and AOF status, last save time and result; see [Persistence](https://docs.hotkv.com/persistence/index.md) |
| `stats` | Connections, commands processed, ops per second, network bytes, expired and evicted keys, keyspace hits and misses, full and partial syncs, ACL denials |
| `replication` | Role, primary link, offsets, connected replicas, backlog; see [Replication](https://docs.hotkv.com/operate/replication/index.md) |
| `cpu` | CPU time used |
| `modules` | Empty: HotKV loads no modules |
| `commandstats` | Per command: `cmdstat_<name>` with calls, total and average microseconds, rejected and failed calls |
| `errorstats` | Per error prefix: `errorstat_<PREFIX>` counts (`ERR`, `WRONGTYPE`, `OOM`, ...) |
| `latencystats` | Per command latency percentiles, `latency_percentiles_usec_<name>` |
| `cluster` | `cluster_enabled`; the section `cluster_info` holds the [`CLUSTER INFO`](https://docs.hotkv.com/commands/cluster-info/index.md) fields. See [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md) |
| `keyspace` | Keys, keys with a TTL, per database |
| `latency` | HotKV's own: `latency_p50`, `latency_p99`, `latency_p999` (microseconds) and the sample count across all commands |

`INFO` with no argument returns the default sections. `INFO all` adds
`commandstats` and `latencystats`, and `INFO everything` also adds HotKV's
`latency` section. You can name several sections at once (`INFO memory
stats`). [`CONFIG RESETSTAT`](https://docs.hotkv.com/commands/config-resetstat/index.md) zeroes the
counters.

Useful fields to alert on:

- `connected_clients` against `maxclients`, and `rejected_connections`
- `used_memory` against `maxmemory`, and `evicted_keys`
- `evicted_clients`, when you set `maxmemory-clients`, and
  `mem_clients_normal` against that limit
- `tracking_total_keys`, for servers whose clients use
  [client-side caching](https://docs.hotkv.com/develop/client-side-caching/index.md#how-many-keys-the-server-remembers)
- `rdb_last_bgsave_status` and `aof_last_write_status` (not `ok` means
  writes may be refused with `MISCONF`)
- `master_link_status` on replicas
- `keyspace_hits` and `keyspace_misses` for a cache's hit rate

## Prometheus metrics

HotKV serves Prometheus metrics over HTTP when `metrics-port` is set:

```
metrics-port 9479
```

or `port = 9479` under `[metrics]` in `hotkv.toml`. It is off by default
(`0`) and is a restart-only setting.

> **Caution**
> The metrics endpoint listens on `127.0.0.1` only. Scrape it from the same
> host, for example with a Prometheus agent or collector running next to
> HotKV, or expose it through a reverse proxy you control.

`GET /metrics` (or `GET /`) returns the text exposition format:

```bash
curl -s http://127.0.0.1:9479/metrics
```

A Prometheus scrape job on the same host:

```yaml
scrape_configs:
  - job_name: hotkv
    static_configs:
      - targets: ["127.0.0.1:9479"]
```

| Series | Type | Meaning |
| --- | --- | --- |
| `hotkv_uptime_seconds` | gauge | Seconds since start |
| `hotkv_connected_clients` | gauge | Connected clients |
| `hotkv_total_connections_received` | counter | Connections accepted |
| `hotkv_rejected_connections` | counter | Connections refused because `maxclients` was reached |
| `hotkv_total_commands_processed` | counter | Commands processed |
| `hotkv_used_memory_bytes` | gauge | Memory in use |
| `hotkv_peak_memory_bytes` | gauge | Peak memory in use |
| `hotkv_keyspace_hits`, `hotkv_keyspace_misses` | counter | Key lookups that found or missed a key |
| `hotkv_expired_keys`, `hotkv_evicted_keys` | counter | Keys expired and evicted |
| `hotkv_db_keys{db}` | gauge | Keys per database |
| `hotkv_command_calls_total{cmd}` | counter | Calls per command |
| `hotkv_command_duration_seconds_avg{cmd}` | gauge | Average duration per command |
| `hotkv_command_duration_seconds_p99{cmd}` | gauge | 99th percentile duration per command |
| `hotkv_workers` | gauge | Worker threads |
| `hotkv_replication_role{role}` | gauge | `primary` or `replica` |
| `hotkv_replication_offset` | counter | Replication offset |
| `hotkv_replication_lag_microseconds` | gauge | Replication lag |
| `hotkv_replication_connected_replicas` | gauge | Connected replicas |

Cluster state (the slots, the nodes, the bus links and the slot migrations) has no
series: see [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md#metrics-endpoint) for what to poll
instead. Enterprise features add their own series when they are in use: the audit
log (`hotkv_audit_entries_total`, `hotkv_audit_file_size_bytes`), encryption
at rest (`hotkv_encryption_enabled`, `hotkv_encryption_provider`), the
semantic and prompt caches (`hotkv_scache_*`, `hotkv_pcache_*`, labelled by
`namespace`) and multi-tenancy (`hotkv_tenant_*`, labelled by tenant).

## Slow log

The slow log keeps the most recent commands that took longer than a
threshold to run (execution time only, not network time), as in Redis.

| Setting | Default | Meaning |
| --- | --- | --- |
| `slowlog-log-slower-than` | `10000` | Threshold in microseconds; `0` logs every command, a negative value none |
| `slowlog-max-len` | `128` | Entries kept; the oldest go first |

Both can be changed with `CONFIG SET`.

```
127.0.0.1:6379> SLOWLOG GET 2
```

[`SLOWLOG GET [count]`](https://docs.hotkv.com/commands/slowlog-get/index.md) returns the newest entries
(10 by default, `-1` for all), each with an id, a Unix timestamp, the
duration in microseconds, the command and its arguments, the client address
and the client name. `SLOWLOG LEN` counts them and `SLOWLOG RESET` clears
them.

`SLOWLOG EXPORT` is a HotKV addition: it returns every entry as a JSON array
that also carries the user, the tenant and the database of each command.

## Latency monitoring

### Latency spikes

Set `latency-monitor-threshold` (milliseconds, default `0` for off) to
record every event that takes at least that long:

```
127.0.0.1:6379> CONFIG SET latency-monitor-threshold 50
OK
```

HotKV records three events: `command` (a slow command), `fast-command` (a
slow command that is normally O(1) or O(log N)) and `expire-cycle` (the
background removal of expired keys). Read them with:

- [`LATENCY LATEST`](https://docs.hotkv.com/commands/latency-latest/index.md): each event's latest and
  worst spike
- [`LATENCY HISTORY event`](https://docs.hotkv.com/commands/latency-history/index.md) and
  `LATENCY GRAPH event`: an event's recent spikes
- [`LATENCY DOCTOR`](https://docs.hotkv.com/commands/latency-doctor/index.md): a plain-text report with
  likely causes
- `LATENCY RESET [event ...]`: forget recorded spikes

### Latency distribution

With `latency-tracking yes` (the default), HotKV keeps a latency
distribution for every command:

- [`LATENCY HISTOGRAM [command ...]`](https://docs.hotkv.com/commands/latency-histogram/index.md) returns
  each command's call count and a histogram in power-of-two microsecond
  buckets.
- `INFO latencystats` reports the percentiles listed in
  `latency-tracking-info-percentiles` (default `50 99 99.9`).

## Watching clients and commands

- [`CLIENT LIST`](https://docs.hotkv.com/commands/client-list/index.md) shows one line per connection,
  with Redis's fields: `id`, `addr`, `name`, `age`, `idle`, `flags`, `db`,
  subscriptions, `qbuf` (input waiting), `omem` (replies not yet read),
  `tot-mem` (what [`maxmemory-clients`](https://docs.hotkv.com/operate/memory-and-eviction/index.md#client-memory)
  counts for the client), `cmd` (last command), `user`, `resp`, `redir`,
  `lib-name` and `lib-ver`. A large `omem` points to a slow consumer. The flags `t`,
  `B` and `R` mark connections that use client-side caching. `CLIENT KILL`
  disconnects one.
- [`MONITOR`](https://docs.hotkv.com/commands/monitor/index.md) streams every command the server runs,
  with its client and database. It is useful for a short look at live
  traffic, but sending every command to the monitoring client costs
  throughput, so do not leave it running on a busy server.
- `INFO commandstats` and `INFO errorstats` show which commands run most,
  which are slowest on average, and which errors clients receive.

## Hot keys

HotKV can count accesses per key and report the most used ones, which helps
find a key that concentrates load (one cluster slot, one cache entry).
Tracking is off by default; turn it on at runtime or in the file:

```
127.0.0.1:6379> CONFIG SET hotkey-tracking yes
OK
```

| Setting | Default | Meaning |
| --- | --- | --- |
| `hotkey-tracking` | `no` | Count accesses per key |
| `hotkey-max-tracked` | `10000` | Most keys tracked at once |

[`HOTKEYS TOP [count]`](https://docs.hotkv.com/commands/hotkeys/index.md) returns the most accessed keys
(10 by default), each with its access count, and `HOTKEYS RESET` clears the
counts. See [Hot keys](https://docs.hotkv.com/develop/hot-keys/index.md) for using it in application
design.

## Enterprise observability

**Availability:** HotKV Cloud: Performance and Scale plans. On-Premise: requires the `enterprise-observability` license feature.

### OBSERVE

[`OBSERVE`](https://docs.hotkv.com/commands/observe/index.md) returns a compact summary as field-value
pairs, for dashboards that poll over the client connection instead of
scraping HTTP:

- `OBSERVE INFO`: uptime, connected clients, total commands, used and peak
  memory, total connections, and the enterprise features the license
  enables.
- `OBSERVE METRICS`: uptime, clients, commands, connections, used and peak
  memory, slow log length, alerts sent and failed, and worker threads.

### ALERTS

[`ALERTS`](https://docs.hotkv.com/commands/alerts/index.md) manages a webhook that HotKV posts alerts to.
Slack, Discord and Telegram webhook URLs are recognised and formatted for
those services; any other URL gets a generic JSON payload.

| Subcommand | What it does |
| --- | --- |
| `ALERTS INFO` | Whether alerting is on, the webhook URL, provider, minimum severity, and delivery counters (`alerts_sent`, `alerts_failed`, `alerts_dropped`, `alerts_rate_limited`, `alerts_deduped`, `alerts_filtered`, `alerts_queued`) |
| `ALERTS SETURL url` | Set the webhook URL (`""` turns alerting off) |
| `ALERTS SETSEVERITY info\|warning\|critical` | Send only alerts at or above this severity |
| `ALERTS ON`, `ALERTS OFF` | Turn sending on or off without forgetting the URL |
| `ALERTS TEST [severity] [message]` | Send a test alert to check delivery |

```
127.0.0.1:6379> ALERTS SETURL https://hooks.slack.com/services/T000/B000/XXXX
OK webhook url updated
127.0.0.1:6379> ALERTS TEST warning Checking the HotKV webhook
OK test alert queued (check ALERTS INFO counters for delivery result)
```

The URL must use `https://`, be at most 1024 characters, and must not
contain a `user:password@` part. URLs pointing to private, loopback,
link-local or cloud metadata addresses are refused unless you set
`block_private_networks = false` under `[alerts]`. `SETURL`, `SETSEVERITY`,
`ON` and `OFF` apply at once and are saved to the `[alerts]` section of
`hotkv.toml`; if saving fails, the reply says the change was applied at
runtime only.

> **Caution**
> `SETURL`, `SETSEVERITY`, `ON` and `OFF` save to the `[alerts]` section of
> `hotkv.toml`. On a server started from `hotkv.conf`, set the `alerts.*`
> settings in the file and restart instead of using these subcommands.

In the configuration file, the webhook is set under `[alerts]` in
`hotkv.toml` (`webhook_url`, `enabled`, `min_severity`, `provider`) or with
`alerts.webhook-url`, `alerts.enabled` and `alerts.min-severity` in
`hotkv.conf`. Sending is rate limited (60 alerts per minute by default) and
repeated alerts of the same kind within 5 seconds are collapsed.

## Next steps

- [Monitoring a cluster](https://docs.hotkv.com/clustering/monitoring/index.md)
- [Troubleshooting](https://docs.hotkv.com/operate/troubleshooting/index.md)
- [Memory and eviction](https://docs.hotkv.com/operate/memory-and-eviction/index.md)
- [Benchmarking](https://docs.hotkv.com/operate/benchmarking/index.md)