# Caching gateway

> A drop-in caching proxy for OpenAI-compatible APIs and Anthropic's Messages API, backed by HotKV. Change the base URL of your app and repeated requests are answered from HotKV instead of being paid for again.

URL: https://docs.hotkv.com/ai/gateway/


`hotkv-gateway` is a caching proxy for OpenAI-compatible APIs, backed by HotKV.
Point your app's `base_url` at the gateway and repeated requests are answered
from HotKV instead of being paid for again. No code changes, in any language.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1")   # the only change
```

For Claude, start the gateway with `--upstream https://api.anthropic.com` and change the same line of
the Anthropic SDK:

```python
import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8080")
```

- **Exact cache.** An identical chat, completion or embeddings request is
  answered from the cache, and so is an identical request to Anthropic's
  Messages API. Streaming requests are captured and replayed as a stream.
- **Semantic cache (optional).** A differently worded question can be answered
  by a cached answer to a similar one.
- **Safe by default.** Each API key has its own cache. Errors and filtered
  answers are never cached. Requests with tool calls, several choices or high
  temperature can be left out.
- **Never in the way.** If HotKV is slow or down, requests go straight to the
  LLM API.
- **Identical requests share one upstream call** while it is in flight.

It works with OpenAI, Azure OpenAI, OpenRouter, vLLM, Ollama, LiteLLM and
anything else that speaks the OpenAI API, and with Anthropic's own API. It
ships as one static binary or a
small container, and it is open source (Apache-2.0):
[github.com/hotkv-ltd/hotkv-gateway](https://github.com/hotkv-ltd/hotkv-gateway).

## Run it

Download a binary from the
[releases page](https://github.com/hotkv-ltd/hotkv-gateway/releases), or
install it with Go:

```bash
go install github.com/hotkv-ltd/hotkv-gateway/cmd/hotkv-gateway@latest
hotkv-gateway --upstream https://api.openai.com --hotkv-url hotkv://localhost:6379
```

The caller's own `Authorization` header goes to the LLM API, so the gateway
needs no key. To keep the key in the gateway instead, set
`HOTKV_GATEWAY_UPSTREAM_API_KEY`. Callers then need none, and they all share
one cache.

Every cacheable response carries an `X-Hotkv-Cache` header: `HIT`,
`SEMANTIC-HIT`, `MISS`, `COALESCED` or `BYPASS`.

## What is cached

| Request | Cached |
| --- | --- |
| `chat/completions`, `completions` and `embeddings` (POST) | yes |
| `/v1/messages`, Anthropic's Messages API (POST) | yes, exact cache only |
| streaming (`"stream": true`) | yes, kept as the event stream and replayed |
| `n` greater than 1 | no |
| `temperature` above `--max-temperature` (off by default) | no |
| a model outside `--models` (all by default) | no |
| any other path (models, `count_tokens`, batches, files, images, audio) | passed through untouched |

A response is stored only when it is a `200` that completed normally: not an
error, not `content_filter`, and for a stream only if it ended with `[DONE]`.

The cache key covers the endpoint, the query string, the caller's credentials
and every body field that can change the answer. `user` and `metadata` do not.
Requests that differ only in key order or number formatting share an entry. The
credentials are the `Authorization`, `Proxy-Authorization`, `X-Api-Key`,
`Api-Key`, `X-Goog-Api-Key`, `OpenAI-Organization`, `OpenAI-Project` and
`Cookie` headers. If your upstream authenticates on another header, name it with
`--tenant-headers`, or callers that differ only in that header would share
answers.

The `anthropic-version`, `anthropic-beta` and `openai-beta` headers are part of
the key too, because a beta feature or an API version can change the answer.

A request header controls the cache for that request. `X-Hotkv-Cache-Control:
no-cache` skips the lookup and refreshes the entry, `no-store` reads but does
not write, and `bypass` skips both. The standard `Cache-Control` header works
the same way.

## Provider prompt caching

OpenAI and Anthropic also cache prompt prefixes on their side. That is a
different thing from this gateway, and the two add up:

- A gateway hit never reaches the provider, so it costs nothing at all.
- A request the gateway has to send upstream still gets the provider's own
  discount for a repeated prefix.

OpenAI does this automatically, so there is nothing to set up. Anthropic needs
`cache_control` in the request. If your app does not set it, start the gateway
with `--anthropic-auto-cache`. Messages API requests that have no
`cache_control` anywhere then get breakpoints on what repeats between requests:
the end of the tool definitions and of the system prompt. Once a conversation
has an earlier assistant turn, it also gets Anthropic's automatic breakpoint,
which follows the conversation as it grows. A single question with no system
prompt and no tools is left alone, because there is nothing to read back later.
Add `--anthropic-cache-ttl 1h` for the one-hour lifetime instead of five
minutes.

Requests that already place their own breakpoints are left alone, and the cache
key is made from what the caller sent, so switching the option never changes
which requests share an entry.

Anthropic bills cache writes above the normal input price, so this pays off when
the same prefix repeats within the lifetime. Leave it off for prompts that are
only sent once.

`/_hotkv/metrics` shows what the providers' own caches did for the requests that
went upstream: `hotkv_gateway_upstream_tokens_total` with `type` set to `input`,
`output`, `cache_read` or `cache_write`.

## Semantic cache

```bash
hotkv-gateway --upstream https://api.openai.com --semantic \
  --embedding-model text-embedding-3-small --semantic-threshold 0.90
```

The gateway embeds the last user message with the upstream's embeddings endpoint
(using the caller's key; the embedding itself is cached) and asks HotKV for the
nearest cached prompt. It needs a license that includes the
[semantic cache](https://docs.hotkv.com/ai-features/index.md#semantic-cache-scache), and the gateway refuses
to start with `--semantic` without one.

The semantic cache works for OpenAI-compatible chat requests. Anthropic has no
embeddings API, so the Messages API uses the exact cache only.

A match by meaning is only safe when everything else that shapes the answer is
the same, so:

- each system prompt and parameter set gets its own cache namespace;
- conversations with earlier user or assistant turns, and requests with tools or
  functions, never use the semantic cache (they still use the exact cache);
- a hit returns the cached answer text as a new response, with zero token counts
  in `usage`.

The default of 0.90 catches close rephrasings. Raise it if you see a wrong
answer served for a different question, lower it to catch looser rephrasings,
and watch the answers either way. An answer the model had to cut short, or that came back empty, is
not kept for semantic reuse; reasoning models need a token limit large enough to
think and then answer.

If the embeddings endpoint fails or takes longer than `--embedding-timeout` (5
seconds), the semantic cache pauses for 30 seconds and requests go on without
it, so a broken embeddings endpoint never slows chat requests down. Azure OpenAI
addresses embeddings through a separate deployment, so use the exact cache
there.

## Licensing

The gateway works on any HotKV server. With a license that includes the
[prompt cache](https://docs.hotkv.com/ai-features/index.md#prompt-cache-pcache), entries live in a `PCACHE`
namespace and HotKV tracks the tokens saved. Without one, it caches under plain
keys and says so at startup. The semantic cache always needs a license.

## Options

| Option | Environment | Default |
| --- | --- | --- |
| `--upstream URL` | `HOTKV_GATEWAY_UPSTREAM` | `https://api.openai.com` |
| `--listen ADDR` | `HOTKV_GATEWAY_LISTEN` | `127.0.0.1:8080` (the container listens on all addresses) |
| `--hotkv-url URL` | `HOTKV_URL` | `hotkv://127.0.0.1:6379`; `hotkvs://` for TLS |
| | `HOTKV_GATEWAY_UPSTREAM_API_KEY` | none |
| `--ttl` | | `1h` |
| `--scope key\|global` | | `key`: a cache per credential. `global` shares answers between callers |
| `--max-temperature N` | | off |
| `--models a,b` | | all |
| `--tenant-headers a,b` | | none |
| `--max-entry-bytes` | | 1 MiB |
| `--max-request-bytes` | | 32 MiB |
| `--max-in-flight N` | | 1024: requests handled at once; more get a 503 |
| `--anthropic-auto-cache` | | off: turn on Anthropic prompt caching for requests that do not use it |
| `--anthropic-cache-ttl 5m\|1h` | | `5m` (with `--anthropic-auto-cache`) |
| `--cache-timeout` | | `250ms`: how long a cache read or write may take before the request goes on without it |
| `--upstream-timeout` | | `10m` |
| `--stream-idle-timeout` | | `5m`: a stream that sends nothing for this long is cut |
| `--semantic`, `--embedding-model`, `--semantic-threshold` | | off, `text-embedding-3-small`, `0.90` |
| `--prefix` | | `gw`: key and namespace prefix on the HotKV server |
| `--log-level` | | `info` |

## Operating it

- `GET /_hotkv/health` returns the cache mode and version. `GET
  /_hotkv/metrics` returns Prometheus text: requests by result, coalesced
  requests, stored entries, upstream and cache errors, and tokens saved by exact
  hits.
- The gateway does not authenticate callers or log bodies, and it passes
  credentials through. Run it on a private network or behind an authenticating
  proxy, and do not expose `/_hotkv/` publicly.
- With `--scope global`, or with a held upstream key, callers share answers. Use
  that only for callers that are allowed to see each other's prompts.
- It starts only if the HotKV server is reachable. Afterwards a failing cache
  never fails a request. It is counted in `hotkv_gateway_cache_errors_total`.
- A stream the upstream breaks halfway is aborted on the caller's side and is
  never stored. A caller that hangs up cancels the upstream call, so unused
  tokens are not generated.
- A cached answer is served to the key that produced it for the whole `--ttl`,
  even if that key was revoked in the meantime, because the gateway does not
  validate keys.
- Cached answers have the model's wording at the time they were stored. Lower
  `--ttl`, or send `no-cache`, when answers must follow a model or prompt
  change.

## Next steps

- [AI features guide](https://docs.hotkv.com/ai-features/index.md)
- [MCP server](https://docs.hotkv.com/ai/mcp-server/index.md)
- [LangGraph checkpointer and store](https://docs.hotkv.com/ai/langgraph/index.md)