Hot keys

All Cloud plansOn-Premise: no license neededCommunity build

A hot key is a key that receives a large share of all traffic: a popular product page, a global counter, a feature flag every request reads. Hot keys are a common cause of uneven load. In a cluster, all requests for one key go to the same node, so a single hot key can saturate that node while the others sit idle. HOTKEYS shows you which keys those are, without sampling tools or a MONITOR session.

Turning tracking on

Hot key tracking is off by default. Turn it on at runtime:

127.0.0.1:6379> CONFIG SET hotkey-tracking yes
OK

or in the configuration file:

hotkey-tracking yes

or in hotkv.toml:

[hotkeys]
tracking = true

While tracking is on, HotKV counts accesses to the first key of every command that names a key (reads and writes alike). Turn it off again with CONFIG SET hotkey-tracking no when you are done diagnosing.

Reading the top keys

HOTKEYS TOP [count] returns the most accessed keys, most frequent first, as pairs of key and access count. count defaults to 10.

127.0.0.1:6379> HOTKEYS TOP 3
1) 1) "product:1187"
   2) (integer) 48211
2) 1) "config:flags"
   2) (integer) 30518
3) 1) "session:9f2c"
   2) (integer) 912

HOTKEYS RESET clears the counts, so you can start a fresh measurement:

127.0.0.1:6379> HOTKEYS RESET
OK

The counts are approximate. Use the numbers to compare keys with each other rather than as exact request totals. Turning tracking off stops the counting but keeps the counts until HOTKEYS RESET; if tracking has never been on, HOTKEYS TOP returns an empty list.

Note

HOTKEYS is an administrative command. A user needs permission for it in their ACL rules (it is in the @admin and @dangerous categories), so grant it to operators rather than to application users.

Python

import hotkv

client = hotkv.HotKV(host="localhost", port=6379)

for key, count in client.hotkeys_top(10):
    print(key, count)
client.hotkeys_reset()

Node.js

import { HotKV } from "@hotkv/client";

const client = new HotKV({ url: "hotkv://127.0.0.1:6379" });

for (const { key, count } of await client.hotKeysTop(10)) {
  console.log(key, count);
}
await client.hotKeysReset();

Go

entries, err := client.HotKeysTop(ctx, 10).Result()
if err != nil {
    log.Fatal(err)
}
for _, e := range entries {
    fmt.Println(e.Key, e.Count)
}

Java

for (var entry : client.hotKeysTop(10)) {
    System.out.println(entry.key() + " " + entry.count());
}

Diagnosing traffic skew

1. Confirm the imbalance. Compare CPU and command rates across nodes, for example with INFO commandstats and cpu on each one. In a cluster, CLUSTER SLOT-STATS shows which slots receive the most traffic.
2. Measure. On the busy node, run HOTKEYS RESET, let traffic run for a minute or two, then run HOTKEYS TOP 20. Each node counts only the keys it serves, so in a cluster ask the node that is under pressure.
3. Read the result. One or two keys far ahead of the rest means a hot key. Counts that fall off gradually across many keys mean the load is simply spread over a busy keyspace, which more nodes or shards will help with.

Fixing a hot key

What helps depends on whether the key is mostly read or mostly written.

Hot reads.

• Cache the value in the application, so most requests never reach HotKV. With client-side caching HotKV tells the application when the key changes, so the copy can be kept until then.
• Keep several copies of the value under different keys (config:flags:0 to config:flags:7, with different hash tags so they land on different nodes) and have each client read a random one. Update every copy when the value changes.
• Send reads to replicas with READONLY if your application tolerates slightly stale data.

Hot writes.

• Split a counter into several keys (views:1187:0 to views:1187:15), increment a random one, and sum them when you read.
• Batch updates in the application and write them less often, for example once a second.

Cache stampedes. When a hot cached key expires, many clients may miss at once and all rebuild it. Let one client rebuild it (for example behind a lock taken with SET lock:key token NX PX 5000) while the others keep serving the old value, or refresh the key shortly before its TTL runs out.

Compatibility

HOTKEYS is a HotKV command; Redis and Valkey have no equivalent built-in command (redis-cli --hotkeys samples keys from the client side instead, and needs an LFU eviction policy).