Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Caching gateway

Last updated

hotkv-gateway is a caching proxy for OpenAI-compatible APIs, backed by HotKV. Point your app’s base_url at the gateway and repeated requests are answered from HotKV instead of being paid for again. No code changes, in any language.

from openai import OpenAI


client = OpenAI(base_url="http://localhost:8080/v1")   # the only change

For Claude, start the gateway with --upstream https://api.anthropic.com and change the same line of the Anthropic SDK:

import anthropic


client = anthropic.Anthropic(base_url="http://localhost:8080")
  • Exact cache. An identical chat, completion or embeddings request is answered from the cache, and so is an identical request to Anthropic’s Messages API. Streaming requests are captured and replayed as a stream.
  • Semantic cache (optional). A differently worded question can be answered by a cached answer to a similar one.
  • Safe by default. Each API key has its own cache. Errors and filtered answers are never cached. Requests with tool calls, several choices or high temperature can be left out.
  • Never in the way. If HotKV is slow or down, requests go straight to the LLM API.
  • Identical requests share one upstream call while it is in flight.

It works with OpenAI, Azure OpenAI, OpenRouter, vLLM, Ollama, LiteLLM and anything else that speaks the OpenAI API, and with Anthropic’s own API. It ships as one static binary or a small container, and it is open source (Apache-2.0): github.com/hotkv-ltd/hotkv-gateway.

Run it

Section titled “Run it”

Download a binary from the releases page, or install it with Go:

Terminal window
go install github.com/hotkv-ltd/hotkv-gateway/cmd/hotkv-gateway@latest
hotkv-gateway --upstream https://api.openai.com --hotkv-url hotkv://localhost:6379

The caller’s own Authorization header goes to the LLM API, so the gateway needs no key. To keep the key in the gateway instead, set HOTKV_GATEWAY_UPSTREAM_API_KEY. Callers then need none, and they all share one cache.

Every cacheable response carries an X-Hotkv-Cache header: HIT, SEMANTIC-HIT, MISS, COALESCED or BYPASS.

What is cached

Section titled “What is cached”
Request Cached
chat/completions, completions and embeddings (POST) yes
/v1/messages, Anthropic’s Messages API (POST) yes, exact cache only
streaming ("stream": true) yes, kept as the event stream and replayed
n greater than 1 no
temperature above --max-temperature (off by default) no
a model outside --models (all by default) no
any other path (models, count_tokens, batches, files, images, audio) passed through untouched

A response is stored only when it is a 200 that completed normally: not an error, not content_filter, and for a stream only if it ended with [DONE].

The cache key covers the endpoint, the query string, the caller’s credentials and every body field that can change the answer. user and metadata do not. Requests that differ only in key order or number formatting share an entry. The credentials are the Authorization, Proxy-Authorization, X-Api-Key, Api-Key, X-Goog-Api-Key, OpenAI-Organization, OpenAI-Project and Cookie headers. If your upstream authenticates on another header, name it with --tenant-headers, or callers that differ only in that header would share answers.

The anthropic-version, anthropic-beta and openai-beta headers are part of the key too, because a beta feature or an API version can change the answer.

A request header controls the cache for that request. X-Hotkv-Cache-Control: no-cache skips the lookup and refreshes the entry, no-store reads but does not write, and bypass skips both. The standard Cache-Control header works the same way.

Provider prompt caching

Section titled “Provider prompt caching”

OpenAI and Anthropic also cache prompt prefixes on their side. That is a different thing from this gateway, and the two add up:

  • A gateway hit never reaches the provider, so it costs nothing at all.
  • A request the gateway has to send upstream still gets the provider’s own discount for a repeated prefix.

OpenAI does this automatically, so there is nothing to set up. Anthropic needs cache_control in the request. If your app does not set it, start the gateway with --anthropic-auto-cache. Messages API requests that have no cache_control anywhere then get breakpoints on what repeats between requests: the end of the tool definitions and of the system prompt. Once a conversation has an earlier assistant turn, it also gets Anthropic’s automatic breakpoint, which follows the conversation as it grows. A single question with no system prompt and no tools is left alone, because there is nothing to read back later. Add --anthropic-cache-ttl 1h for the one-hour lifetime instead of five minutes.

Requests that already place their own breakpoints are left alone, and the cache key is made from what the caller sent, so switching the option never changes which requests share an entry.

Anthropic bills cache writes above the normal input price, so this pays off when the same prefix repeats within the lifetime. Leave it off for prompts that are only sent once.

/_hotkv/metrics shows what the providers’ own caches did for the requests that went upstream: hotkv_gateway_upstream_tokens_total with type set to input, output, cache_read or cache_write.

Semantic cache

Section titled “Semantic cache”
Terminal window
hotkv-gateway --upstream https://api.openai.com --semantic \
  --embedding-model text-embedding-3-small --semantic-threshold 0.90

The gateway embeds the last user message with the upstream’s embeddings endpoint (using the caller’s key; the embedding itself is cached) and asks HotKV for the nearest cached prompt. It needs a license that includes the semantic cache, and the gateway refuses to start with --semantic without one.

The semantic cache works for OpenAI-compatible chat requests. Anthropic has no embeddings API, so the Messages API uses the exact cache only.

A match by meaning is only safe when everything else that shapes the answer is the same, so:

  • each system prompt and parameter set gets its own cache namespace;
  • conversations with earlier user or assistant turns, and requests with tools or functions, never use the semantic cache (they still use the exact cache);
  • a hit returns the cached answer text as a new response, with zero token counts in usage.

The default of 0.90 catches close rephrasings. Raise it if you see a wrong answer served for a different question, lower it to catch looser rephrasings, and watch the answers either way. An answer the model had to cut short, or that came back empty, is not kept for semantic reuse; reasoning models need a token limit large enough to think and then answer.

If the embeddings endpoint fails or takes longer than --embedding-timeout (5 seconds), the semantic cache pauses for 30 seconds and requests go on without it, so a broken embeddings endpoint never slows chat requests down. Azure OpenAI addresses embeddings through a separate deployment, so use the exact cache there.

Licensing

Section titled “Licensing”

The gateway works on any HotKV server. With a license that includes the prompt cache, entries live in a PCACHE namespace and HotKV tracks the tokens saved. Without one, it caches under plain keys and says so at startup. The semantic cache always needs a license.

Options

Section titled “Options”
Option Environment Default
--upstream URL HOTKV_GATEWAY_UPSTREAM https://api.openai.com
--listen ADDR HOTKV_GATEWAY_LISTEN 127.0.0.1:8080 (the container listens on all addresses)
--hotkv-url URL HOTKV_URL hotkv://127.0.0.1:6379; hotkvs:// for TLS
HOTKV_GATEWAY_UPSTREAM_API_KEY none
--ttl 1h
--scope key|global key: a cache per credential. global shares answers between callers
--max-temperature N off
--models a,b all
--tenant-headers a,b none
--max-entry-bytes 1 MiB
--max-request-bytes 32 MiB
--max-in-flight N 1024: requests handled at once; more get a 503
--anthropic-auto-cache off: turn on Anthropic prompt caching for requests that do not use it
--anthropic-cache-ttl 5m|1h 5m (with --anthropic-auto-cache)
--cache-timeout 250ms: how long a cache read or write may take before the request goes on without it
--upstream-timeout 10m
--stream-idle-timeout 5m: a stream that sends nothing for this long is cut
--semantic, --embedding-model, --semantic-threshold off, text-embedding-3-small, 0.90
--prefix gw: key and namespace prefix on the HotKV server
--log-level info

Operating it

Section titled “Operating it”
  • GET /_hotkv/health returns the cache mode and version. GET /_hotkv/metrics returns Prometheus text: requests by result, coalesced requests, stored entries, upstream and cache errors, and tokens saved by exact hits.
  • The gateway does not authenticate callers or log bodies, and it passes credentials through. Run it on a private network or behind an authenticating proxy, and do not expose /_hotkv/ publicly.
  • With --scope global, or with a held upstream key, callers share answers. Use that only for callers that are allowed to see each other’s prompts.
  • It starts only if the HotKV server is reachable. Afterwards a failing cache never fails a request. It is counted in hotkv_gateway_cache_errors_total.
  • A stream the upstream breaks halfway is aborted on the caller’s side and is never stored. A caller that hangs up cancels the upstream call, so unused tokens are not generated.
  • A cached answer is served to the key that produced it for the whole --ttl, even if that key was revoked in the meantime, because the gateway does not validate keys.
  • Cached answers have the model’s wording at the time they were stored. Lower --ttl, or send no-cache, when answers must follow a model or prompt change.

Next steps

Section titled “Next steps”