Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Monitoring

Last updated

HotKV exposes the same monitoring surface as Redis (INFO, SLOWLOG, LATENCY, CLIENT LIST, MONITOR), plus a Prometheus endpoint, hot key tracking, and enterprise observability commands. This page covers what each one tells you and how to turn it on.

INFO

Section titled “INFO”

INFO returns the server’s state as text, in sections. It keeps Redis’s section names, field names and order, so tools written against Redis’s INFO find the fields where they expect them.

127.0.0.1:6379> INFO memory
Section What it covers
server Version (redis_version for tool compatibility, and hotkv_version), process, uptime, ports, config file, worker threads
clients Connected and blocked clients, maxclients, largest input and output buffers
memory used_memory, RSS, peak, maxmemory, policy, fragmentation; see Memory and eviction
persistence Snapshot and AOF status, last save time and result; see Persistence
stats Connections, commands processed, ops per second, network bytes, expired and evicted keys, keyspace hits and misses, full and partial syncs, ACL denials
replication Role, primary link, offsets, connected replicas, backlog; see Replication
cpu CPU time used
modules Empty: HotKV loads no modules
commandstats Per command: cmdstat_<name> with calls, total and average microseconds, rejected and failed calls
errorstats Per error prefix: errorstat_<PREFIX> counts (ERR, WRONGTYPE, OOM, …)
latencystats Per command latency percentiles, latency_percentiles_usec_<name>
cluster cluster_enabled; the section cluster_info holds the CLUSTER INFO fields. See Monitoring a cluster
keyspace Keys, keys with a TTL, per database
latency HotKV’s own: latency_p50, latency_p99, latency_p999 (microseconds) and the sample count across all commands

INFO with no argument returns the default sections. INFO all adds commandstats and latencystats, and INFO everything also adds HotKV’s latency section. You can name several sections at once (INFO memory stats). CONFIG RESETSTAT zeroes the counters.

Useful fields to alert on:

  • connected_clients against maxclients, and rejected_connections
  • used_memory against maxmemory, and evicted_keys
  • evicted_clients, when you set maxmemory-clients, and mem_clients_normal against that limit
  • tracking_total_keys, for servers whose clients use client-side caching
  • rdb_last_bgsave_status and aof_last_write_status (not ok means writes may be refused with MISCONF)
  • master_link_status on replicas
  • keyspace_hits and keyspace_misses for a cache’s hit rate

Prometheus metrics

Section titled “Prometheus metrics”

HotKV serves Prometheus metrics over HTTP when metrics-port is set:

metrics-port 9479

or port = 9479 under [metrics] in hotkv.toml. It is off by default (0) and is a restart-only setting.

GET /metrics (or GET /) returns the text exposition format:

Terminal window
curl -s http://127.0.0.1:9479/metrics

A Prometheus scrape job on the same host:

scrape_configs:
  - job_name: hotkv
    static_configs:
      - targets: ["127.0.0.1:9479"]
Series Type Meaning
hotkv_uptime_seconds gauge Seconds since start
hotkv_connected_clients gauge Connected clients
hotkv_total_connections_received counter Connections accepted
hotkv_rejected_connections counter Connections refused because maxclients was reached
hotkv_total_commands_processed counter Commands processed
hotkv_used_memory_bytes gauge Memory in use
hotkv_peak_memory_bytes gauge Peak memory in use
hotkv_keyspace_hits, hotkv_keyspace_misses counter Key lookups that found or missed a key
hotkv_expired_keys, hotkv_evicted_keys counter Keys expired and evicted
hotkv_db_keys{db} gauge Keys per database
hotkv_command_calls_total{cmd} counter Calls per command
hotkv_command_duration_seconds_avg{cmd} gauge Average duration per command
hotkv_command_duration_seconds_p99{cmd} gauge 99th percentile duration per command
hotkv_workers gauge Worker threads
hotkv_replication_role{role} gauge primary or replica
hotkv_replication_offset counter Replication offset
hotkv_replication_lag_microseconds gauge Replication lag
hotkv_replication_connected_replicas gauge Connected replicas

Cluster state (the slots, the nodes, the bus links and the slot migrations) has no series: see Monitoring a cluster for what to poll instead. Enterprise features add their own series when they are in use: the audit log (hotkv_audit_entries_total, hotkv_audit_file_size_bytes), encryption at rest (hotkv_encryption_enabled, hotkv_encryption_provider), the semantic and prompt caches (hotkv_scache_*, hotkv_pcache_*, labelled by namespace) and multi-tenancy (hotkv_tenant_*, labelled by tenant).

Slow log

Section titled “Slow log”

The slow log keeps the most recent commands that took longer than a threshold to run (execution time only, not network time), as in Redis.

Setting Default Meaning
slowlog-log-slower-than 10000 Threshold in microseconds; 0 logs every command, a negative value none
slowlog-max-len 128 Entries kept; the oldest go first

Both can be changed with CONFIG SET.

127.0.0.1:6379> SLOWLOG GET 2

SLOWLOG GET [count] returns the newest entries (10 by default, -1 for all), each with an id, a Unix timestamp, the duration in microseconds, the command and its arguments, the client address and the client name. SLOWLOG LEN counts them and SLOWLOG RESET clears them.

SLOWLOG EXPORT is a HotKV addition: it returns every entry as a JSON array that also carries the user, the tenant and the database of each command.

Latency monitoring

Section titled “Latency monitoring”

Latency spikes

Section titled “Latency spikes”

Set latency-monitor-threshold (milliseconds, default 0 for off) to record every event that takes at least that long:

127.0.0.1:6379> CONFIG SET latency-monitor-threshold 50
OK

HotKV records three events: command (a slow command), fast-command (a slow command that is normally O(1) or O(log N)) and expire-cycle (the background removal of expired keys). Read them with:

Latency distribution

Section titled “Latency distribution”

With latency-tracking yes (the default), HotKV keeps a latency distribution for every command:

  • LATENCY HISTOGRAM [command ...] returns each command’s call count and a histogram in power-of-two microsecond buckets.
  • INFO latencystats reports the percentiles listed in latency-tracking-info-percentiles (default 50 99 99.9).

Watching clients and commands

Section titled “Watching clients and commands”
  • CLIENT LIST shows one line per connection, with Redis’s fields: id, addr, name, age, idle, flags, db, subscriptions, qbuf (input waiting), omem (replies not yet read), tot-mem (what maxmemory-clients counts for the client), cmd (last command), user, resp, redir, lib-name and lib-ver. A large omem points to a slow consumer. The flags t, B and R mark connections that use client-side caching. CLIENT KILL disconnects one.
  • MONITOR streams every command the server runs, with its client and database. It is useful for a short look at live traffic, but sending every command to the monitoring client costs throughput, so do not leave it running on a busy server.
  • INFO commandstats and INFO errorstats show which commands run most, which are slowest on average, and which errors clients receive.

Hot keys

Section titled “Hot keys”

HotKV can count accesses per key and report the most used ones, which helps find a key that concentrates load (one cluster slot, one cache entry). Tracking is off by default; turn it on at runtime or in the file:

127.0.0.1:6379> CONFIG SET hotkey-tracking yes
OK
Setting Default Meaning
hotkey-tracking no Count accesses per key
hotkey-max-tracked 10000 Most keys tracked at once

HOTKEYS TOP [count] returns the most accessed keys (10 by default), each with its access count, and HOTKEYS RESET clears the counts. See Hot keys for using it in application design.

Enterprise observability

Section titled “Enterprise observability”
Cloud: Performance and ScaleOn-Premise: enterprise-observability license

OBSERVE

Section titled “OBSERVE”

OBSERVE returns a compact summary as field-value pairs, for dashboards that poll over the client connection instead of scraping HTTP:

  • OBSERVE INFO: uptime, connected clients, total commands, used and peak memory, total connections, and the enterprise features the license enables.
  • OBSERVE METRICS: uptime, clients, commands, connections, used and peak memory, slow log length, alerts sent and failed, and worker threads.

ALERTS

Section titled “ALERTS”

ALERTS manages a webhook that HotKV posts alerts to. Slack, Discord and Telegram webhook URLs are recognised and formatted for those services; any other URL gets a generic JSON payload.

Subcommand What it does
ALERTS INFO Whether alerting is on, the webhook URL, provider, minimum severity, and delivery counters (alerts_sent, alerts_failed, alerts_dropped, alerts_rate_limited, alerts_deduped, alerts_filtered, alerts_queued)
ALERTS SETURL url Set the webhook URL ("" turns alerting off)
ALERTS SETSEVERITY info|warning|critical Send only alerts at or above this severity
ALERTS ON, ALERTS OFF Turn sending on or off without forgetting the URL
ALERTS TEST [severity] [message] Send a test alert to check delivery 
127.0.0.1:6379> ALERTS SETURL https://hooks.slack.com/services/T000/B000/XXXX
OK webhook url updated
127.0.0.1:6379> ALERTS TEST warning Checking the HotKV webhook
OK test alert queued (check ALERTS INFO counters for delivery result)

The URL must use https://, be at most 1024 characters, and must not contain a user:password@ part. URLs pointing to private, loopback, link-local or cloud metadata addresses are refused unless you set block_private_networks = false under [alerts]. SETURL, SETSEVERITY, ON and OFF apply at once and are saved to the [alerts] section of hotkv.toml; if saving fails, the reply says the change was applied at runtime only.

In the configuration file, the webhook is set under [alerts] in hotkv.toml (webhook_url, enabled, min_severity, provider) or with alerts.webhook-url, alerts.enabled and alerts.min-severity in hotkv.conf. Sending is rate limited (60 alerts per minute by default) and repeated alerts of the same kind within 5 seconds are collapsed.

Next steps

Section titled “Next steps”