Client-side caching

A read answered from your application’s own memory takes no network round trip. The hard part is knowing when a copy has gone stale. With server-assisted client-side caching, a connection asks HotKV to remember which keys it reads, and HotKV sends it an invalidation when one of them changes. The application keeps its copy until that message arrives.

You turn it on for one connection with CLIENT TRACKING. A connection that does not use it is not affected.

How it works

1. The client reads a key with an ordinary read command such as GET. HotKV notes that this connection read the key, and a change made at the same moment is never missed.
2. The client keeps the value in its local cache.
3. When the key next changes, HotKV sends the client an invalidation and forgets the note. The client drops the key from its cache.
4. The next time the client reads the key it is noted again, so the client is told of the next change, and only then.

A key is told once for each time it was read. A client that wants to hear about a key it still uses must read it again after each invalidation.

Try it with redis-cli

A RESP2 connection cannot receive messages it did not ask for, so it sends its invalidations to a second connection that is subscribed to a channel. Open three redis-cli sessions.

The listener subscribes to the channel HotKV publishes invalidations on and tells you its connection id:

127.0.0.1:6379> CLIENT ID
(integer) 52
127.0.0.1:6379> SUBSCRIBE __redis__:invalidate
1) "subscribe"
2) "__redis__:invalidate"
3) (integer) 1

The reader turns tracking on, redirecting its invalidations to the listener, and reads a key:

127.0.0.1:6379> CLIENT TRACKING ON REDIRECT 52
OK
127.0.0.1:6379> GET user:1
(nil)

A third session changes the key:

127.0.0.1:6379> SET user:1 bob
OK

The listener is sent the invalidation:

1) "message"
2) "__redis__:invalidate"
3) 1) "user:1"

The reader read user:1 before it existed. A key that is missing is noted too, so the client is told when it appears.

RESP3: one connection

A RESP3 connection (after HELLO 3) can be pushed to, so it needs no second connection. In the transcripts below, a line that starts with push> is a message the server sent on its own, after the reply of the command that caused it. Client libraries receive it as a push frame, >2, then invalidate, then an array of keys.

# Connection A (the reply of HELLO 3 is left out)
127.0.0.1:6379> HELLO 3
127.0.0.1:6379> CLIENT TRACKING ON
OK
127.0.0.1:6379> SET user:1 alice
OK
127.0.0.1:6379> MGET user:1 user:2
1) "alice"
2) (nil)

# Connection B
127.0.0.1:6379> SET user:1 bob
OK

# Connection A is sent, without asking:
push> 1) "invalidate"
      2) 1) "user:1"

Connection A read both keys, so it is told of either. It is not told again when user:1 changes a second time, because it has not read the key since. When user:2 is created it is told:

# Connection B
127.0.0.1:6379> SET user:1 carol
OK
127.0.0.1:6379> SET user:2 dave
OK

# Connection A
push> 1) "invalidate"
      2) 1) "user:2"

A change that empties a whole database is told with a null in place of the keys, which means “forget everything you cached”:

# Connection B
127.0.0.1:6379> FLUSHDB
OK

# Connection A
push> 1) "invalidate"
      2) (nil)

Modes

CLIENT TRACKING ON takes options that decide which keys a client is told of.

Mode	The client is told of
Default	The keys it read.
OPTIN	The keys read by a command that comes right after CLIENT CACHING YES.
OPTOUT	The keys it read, except those of a command that comes right after CLIENT CACHING NO.
BCAST	Every key that starts with one of its PREFIX values, whether it read the key or not. HotKV remembers nothing per key for this client.

NOLOOP, in any mode, leaves out the changes made by the client’s own commands: it updates its cache itself when it writes. REDIRECT client-id sends the invalidations to another connection (see Where invalidations go).

OPTIN and OPTOUT cannot be used together, or with BCAST. A PREFIX needs BCAST. Prefixes of one client must not overlap: one being the start of another, or giving the same one twice, is refused with an error that names both. Once tracking is on, BCAST cannot be switched on or off, and neither can OPTIN for OPTOUT: turn tracking off first. Asking for CLIENT TRACKING ON again replaces the redirect and the other options, and adds any new prefixes.

OPTIN and OPTOUT

CLIENT CACHING chooses for the next command only. Sent before MULTI, it covers the commands queued up to EXEC.

# Connection A, tracking in OPTIN mode
127.0.0.1:6379> CLIENT TRACKING ON OPTIN
OK
127.0.0.1:6379> GET p:1
"1"
127.0.0.1:6379> CLIENT CACHING YES
OK
127.0.0.1:6379> GET p:2
"2"

# Connection B changes both keys
127.0.0.1:6379> SET p:1 changed
OK
127.0.0.1:6379> SET p:2 changed
OK

# Connection A is told only of the key it opted in for
push> 1) "invalidate"
      2) 1) "p:2"

In OPTOUT mode the choice is the other way round: CLIENT CACHING NO before a command leaves its keys out.

BCAST

A client in BCAST mode does not need to read a key to hear about it, which suits a cache that holds everything under a prefix. The client is told of the keys one command changed in one message for each prefix:

# Connection A
127.0.0.1:6379> CLIENT TRACKING ON BCAST PREFIX user: PREFIX order:
OK

# Connection B
127.0.0.1:6379> MSET user:1 x user:2 y order:9 z other:1 w
OK

# Connection A
push> 1) "invalidate"
      2) 1) "user:1"
         2) "user:2"
push> 1) "invalidate"
      2) 1) "order:9"

CLIENT TRACKING ON BCAST with no prefix covers every key. A client is told only of the keys its ACL user may read.

NOLOOP

# Connection A
127.0.0.1:6379> CLIENT TRACKING ON BCAST PREFIX n: NOLOOP
OK
127.0.0.1:6379> SET n:1 mine
OK

# Connection B
127.0.0.1:6379> SET n:1 theirs
OK

# Connection A is told of the change by B only
push> 1) "invalidate"
      2) 1) "n:1"

What counts as a read, and what as a change

Reads. A key is noted when it is named by a command that only reads: GET, MGET, HGETALL, EXISTS, TTL and the like. A key that does not exist is noted too. Notes are made as each command runs, so inside MULTI and EXEC every read command notes its keys, and a script notes the keys of the read commands it calls, for the client that ran the script. EVAL, EVALSHA and FCALL note nothing themselves. Commands that only write note nothing. The note is made by key name whatever database the client had selected, so an invalidation can arrive for a key of another database; the client may be told more than it needs, and never less.

Changes. A key is told as changed when:

• a command that writes it has run, whether or not the command altered the value (inside MULTI and scripts too). This includes changing its time to live;
• the key expires, or the fields of a hash expire (a client that reads a key just as its time runs out can be told of the expiry right after the reply);
• it is evicted to stay under maxmemory;
• it is moved by MIGRATE without COPY, or created by RESTORE;
• its slot is cleared or taken away (CLUSTER FLUSHSLOT, SFLUSH, the end of a slot migration);
• it is written on a replica by the stream from its primary, for the clients connected to that replica.

FLUSHDB, FLUSHALL, SWAPDB, CLUSTER RESET of a replica, CLUSTER REPLICATE NO ONE and a replica’s data being replaced by a full sync tell every client that tracks with a null instead of keys.

The data the AI commands keep outside the keyspace (for example the entries of a prompt cache) is not noted and not told.

Where invalidations go

On a RESP3 connection the invalidation is a push on the same connection. It comes after the reply of the command that caused it, and whatever CLIENT REPLY says. A client’s own change is told once its reply is out, and the changes of a transaction come after the EXEC reply. Evictions that make room for a command are told before that command’s reply.

A RESP2 connection cannot receive a push. With REDIRECT its invalidations go to the other connection: as a push if that one speaks RESP3, otherwise as a message on the channel __redis__:invalidate, provided that connection is subscribed to something. A RESP2 client with no redirect, or one that redirects to a connection that is not subscribed, is told nothing, though CLIENT TRACKING ON is accepted. The target of a REDIRECT must exist when the command runs.

127.0.0.1:6379> CLIENT TRACKING ON REDIRECT 99999
(error) ERR The client ID you want redirect to does not exist

When the connection redirected to is gone, a RESP3 client is sent tracking-redir-broken with that connection’s id, each time an invalidation would have gone to it:

push> 1) "tracking-redir-broken"
      2) (integer) 48

CLIENT TRACKINGINFO then lists broken_redirect among the flags until you turn tracking on again.

A RESP3 connection that is subscribed to a channel can still run any command, so a single connection can hold tracking, subscriptions and ordinary reads. See Pub/Sub.

What your cache must do

• Drop the key on an invalidation, and read it again if you still want it. That read is what makes the server tell you about the next change.
• Treat a null invalidation as “forget everything”.
• Empty the cache when the connection is lost. Changes made while you were disconnected were not delivered. Turn tracking on again after you reconnect.
• Watch for tracking-redir-broken if you redirect. Your cache is no longer being kept current: empty it and set up the redirect again.
• Keep reads and the cache consistent. Put a value in the cache only for a read made on the connection that has tracking on, and do not let a value in flight overwrite an invalidation that arrived meanwhile.

How many keys the server remembers

The server remembers which keys tracking clients have read. Four INFO fields describe it:

Field	Section	Meaning
tracking_clients	clients	The connections with tracking on.
tracking_total_keys	stats	The keys that have a note.
tracking_total_items	stats	The notes: a key read by two clients counts twice.
tracking_total_prefixes	stats	The prefixes BCAST clients registered.

tracking-table-max-keys limits the keys in the table (1,000,000 by default; 0 means no limit). Past it, HotKV forgets keys, in no particular order, and tells each client that had read one as if the key had changed, so a cache is never left holding a value the server stopped watching. CONFIG SET applies a new limit at once. With a limit of 3 and a client that read five keys:

127.0.0.1:6379> CONFIG SET tracking-table-max-keys 3
OK
127.0.0.1:6379> MGET k1 k2 k3 k4 k5
1) (nil)
2) (nil)
3) (nil)
4) (nil)
5) (nil)
push> 1) "invalidate"
      2) 1) "k2"
push> 1) "invalidate"
      2) 1) "k4"

A client that is told of a key it just read has to read it again; if you see this often, raise the limit or use BCAST with prefixes, which keeps nothing per key. A connection that closes leaves its notes in the table until the key changes or is forgotten.

Memory and limits

Invalidations count in the output of the connection they are sent to, like any other message. A client that does not read them is closed by client-output-buffer-limit or evicted by maxmemory-clients, as any client that does not read its replies. A client with no hard output limit (the normal class, by default) is also closed when it leaves more than 32 MiB of pushes unread. Prefixes a BCAST client registers count towards that client’s memory (tot-mem in CLIENT LIST). See Client memory.

In a cluster and on replicas

Every node keeps its own table, and a client is told of the changes the node it reads from makes. For a key in a slot that node serves, that is every change. A client that reads from a replica (after READONLY) is told of the writes the replica applies from its primary. When Redis 8’s atomic slot migration leaves slots on the node they came from, each removed key is told to the clients that track it. A client that follows a slot to another node should treat the move as a reason to drop what it cached from the old one. See Cluster clients.

Seeing what is tracked

CLIENT TRACKINGINFO shows the state of the current connection, and CLIENT LIST shows t (tracking is on), B (BCAST) and R (the redirect is broken) among a connection’s flags and redir= for the connection invalidations go to:

127.0.0.1:6379> CLIENT TRACKINGINFO
1) "flags"
2) 1) "on"
   2) "bcast"
3) "redirect"
4) (integer) 0
5) "prefixes"
6) 1) "order:"
   2) "user:"

From a client library

Redis client libraries that have a local cache turn tracking on for you and handle the invalidations. The examples below use the redis packages, which work against HotKV like any Redis-compatible client (see Connecting and SDKs).

Python

As of redis-py 8.1.0 the built-in cache (cache_config) connects only to a server whose HELLO reply says it is Redis, so it refuses a HotKV connection. This cache manages tracking itself, with a RESP2 listener connection and REDIRECT, which works with any client library:

import threading
import time

import redis

HOST, PORT = "localhost", 6379


class TrackedCache:
    """A read-through cache that the server keeps current."""

    def __init__(self):
        self.local = {}
        self.lock = threading.Lock()
        self.hits = 0

        # This connection only listens. Its invalidations arrive as
        # messages on a channel because the connection speaks RESP2.
        self.pubsub = redis.Redis(
            host=HOST, port=PORT, decode_responses=True, protocol=2
        ).pubsub()
        self.pubsub.execute_command("CLIENT", "ID")
        listener_id = self.pubsub.parse_response()
        self.pubsub.subscribe("__redis__:invalidate")
        self.pubsub.get_message(timeout=5)  # the subscribe confirmation

        # This connection reads. The pool holds one connection so that
        # every read goes through the one that has tracking turned on.
        self.reader = redis.Redis(
            host=HOST, port=PORT, decode_responses=True, protocol=2,
            max_connections=1,
        )
        self.reader.execute_command(
            "CLIENT", "TRACKING", "ON", "REDIRECT", listener_id
        )

        threading.Thread(target=self._listen, daemon=True).start()

    def _listen(self):
        for message in self.pubsub.listen():
            if message["type"] != "message":
                continue
            with self.lock:
                if message["data"] is None:  # a flush: forget everything
                    self.local.clear()
                else:
                    for key in message["data"]:
                        self.local.pop(key, None)

    def get(self, key):
        with self.lock:
            if key in self.local:
                self.hits += 1
                return self.local[key]
        value = self.reader.get(key)
        with self.lock:
            self.local[key] = value
        return value


cache = TrackedCache()
writer = redis.Redis(host=HOST, port=PORT, decode_responses=True)
writer.set("user:1", "alice")

print(cache.get("user:1"), "local hits:", cache.hits)
print(cache.get("user:1"), "local hits:", cache.hits)
writer.set("user:1", "bob")
time.sleep(0.2)  # let the invalidation arrive
print(cache.get("user:1"), "local hits:", cache.hits)
print(cache.get("user:1"), "local hits:", cache.hits)

Run against a server, this prints:

alice local hits: 0
alice local hits: 1
bob local hits: 1
bob local hits: 2

The second read is answered locally. After the write, the invalidation removes the entry, so the next read goes to the server and the one after it is local again.

Node.js

node-redis turns on tracking and keeps the cache when you give createClient RESP3 and a clientSideCache:

import { createClient } from 'redis';

const url = 'redis://localhost:6379';

const writer = createClient({ url });
await writer.connect();

const cached = createClient({
  url,
  RESP: 3,
  clientSideCache: { ttl: 0, maxEntries: 1000, evictPolicy: 'LRU' },
});
await cached.connect();

await writer.set('user:1', 'alice');
console.log(await cached.get('user:1')); // asks the server
console.log(await cached.get('user:1')); // answered from the local cache

await writer.set('user:1', 'bob');
await new Promise((resolve) => setTimeout(resolve, 200)); // let the invalidation arrive
console.log(await cached.get('user:1')); // asks the server again
console.log(await cached.get('user:1')); // local again

console.log(cached.clientSideCache.stats());
await cached.quit();
await writer.quit();

Run against a server, this prints:

alice
alice
bob
bob
CacheStats {
  hitCount: 2,
  missCount: 2,
  loadSuccessCount: 2,
  loadFailureCount: 0,
  totalLoadTime: 60.73290000000043,
  evictionCount: 0
}

totalLoadTime is milliseconds and differs from run to run. The server saw two GET commands for the four reads.

Compatibility

CLIENT TRACKING, CLIENT CACHING, CLIENT GETREDIR and CLIENT TRACKINGINFO take the options, give the replies and refuse with the error texts that Redis 8 and Valkey 9 do, on RESP2 and RESP3, so a client library written for their client-side caching works, apart from the check some libraries make of the server’s name (see the Python example above). The HotKV SDKs do not include a local cache; use a Redis client library that does, as above.

Things to keep in mind:

• Expiry is told like any other change, for a key and for the fields of a hash.
• tracking-table-max-keys forgets keys in no particular order.
• Tracking is per node: invalidations come from the node the client reads from.

See also

CLIENT TRACKING, CLIENT CACHING, CLIENT GETREDIR, CLIENT TRACKINGINFO, RESP3, Pub/Sub, Keyspace notifications