# Benchmarking

> Benchmark HotKV fairly with redis-benchmark and memtier_benchmark, with enough client load, pipelining, a filled keyspace and randomized keys.

URL: https://docs.hotkv.com/operate/benchmarking/


HotKV speaks the Redis protocol, so the standard tools work against it
unchanged: `redis-benchmark` and `memtier_benchmark`. This page is about
getting numbers you can trust, whether you are sizing a deployment or
comparing HotKV with Redis or Valkey. HotKV's own published results are on
the [Performance](https://docs.hotkv.com/performance/index.md) page.

## What makes a benchmark fair

Most misleading benchmark results come from the setup, not the server. Before
you compare numbers:

- **Run the load generator on a different machine** from the server, on the
  same network you will use in production. A client on the server's host
  takes CPU from the server and skips the network entirely.
- **Make sure the client is not the bottleneck.** Give the load generator
  enough cores and threads, and watch its CPU during the run. If the client
  machine is at 100% CPU, you are measuring the client. Use several client
  machines if needed.
- **Know your network ceiling.** At high request rates or large values, the
  network link limits throughput before the server does. Check the
  bandwidth between the machines (for example with `iperf3`) and compare it
  with the bytes per second the benchmark moves.
- **Use the same hardware and settings for every server you compare**:
  same machine type, same persistence settings (all off, or all on with the
  same fsync policy), same `maxmemory` and eviction policy, same password or
  TLS settings.
- **Give every server its threading.** HotKV uses all CPU cores by default
  (`threads 0`). Redis and Valkey need `io-threads` set to use more than one
  core for network I/O. Configure each server the way you would run it in
  production, and write down what you used.
- **Fill the keyspace before measuring reads.** A `GET` on a key that does
  not exist returns nil without touching any data, so a read test against an
  empty or partly filled server measures misses. Load every key first, then
  check `DBSIZE`.
- **Randomize key choice.** Some tools use a fixed random seed by default, so
  every client requests the same sequence of keys. Use per-client random
  seeds so the access pattern looks like real traffic.
- **Run long enough, and more than once.** Discard a warm-up period, run for
  at least a minute, repeat each test a few times, and report the median.
  Report latency percentiles (p50, p99, p99.9) alongside throughput.
- **State the pipeline depth.** Results with pipelining and without are not
  comparable; report both if your application uses both.

> **Tip**
> Run `CONFIG RESETSTAT` on the server before each test. Afterwards,
> `INFO commandstats` shows the calls and server-side time per command, and
> `INFO stats` shows `keyspace_hits` and `keyspace_misses`: a read test
> should show hits, not misses.

## memtier_benchmark

`memtier_benchmark` gives fine control over key patterns, ratios and
client concurrency.

**1. Fill the keyspace** with every key the read test will use (here
1,000,000 keys of 64 bytes):

```bash
memtier_benchmark -s 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  --ratio=1:0 --key-pattern=P:P \
  --key-minimum=1 --key-maximum=1000000 \
  --data-size=64 --requests=allkeys \
  --threads=4 --clients=10 --hide-histogram
```

Check the result with `redis-cli -h 10.0.0.1 -a "$HOTKV_PASSWORD" DBSIZE`.

**2. Run the test** with random keys over the same range, a different
random seed per client, and a fixed duration:

```bash
memtier_benchmark -s 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  --ratio=1:10 --key-pattern=R:R \
  --key-minimum=1 --key-maximum=1000000 \
  --data-size=64 --test-time=60 \
  --threads=8 --clients=50 --pipeline=1 \
  --randomize --distinct-client-seed
```

| Option | What it controls |
| --- | --- |
| `--threads`, `--clients` | Load generator threads, and connections per thread. Raise them until throughput stops growing or latency climbs. |
| `--pipeline` | Requests in flight per connection. `1` is no pipelining. |
| `--ratio=SET:GET` | The write to read mix (`1:0` writes only, `0:1` reads only) |
| `--key-pattern` | `P:P` walks the range in parallel blocks (for filling), `R:R` picks keys at random, `G:G` concentrates on the middle of the range |
| `--randomize`, `--distinct-client-seed` | A random seed for the run, and a different one per client |
| `--data-size` | Value size in bytes |
| `--run-count` | Repeat the test and report each run |
| `--cluster-mode` | Follow cluster redirects when testing a [cluster](https://docs.hotkv.com/clustering/index.md) |

## redis-benchmark

`redis-benchmark` is quick to run and covers many commands in one go:

```bash
redis-benchmark -h 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  -t set,get -n 10000000 -r 1000000 -d 64 \
  -c 200 --threads 8 -P 16 -q
```

| Option | What it controls |
| --- | --- |
| `-c` | Total connections |
| `--threads` | Load generator threads; without it, one thread drives every connection and usually becomes the limit |
| `-P` | Pipeline depth (default `1`, no pipelining) |
| `-r` | Use random keys from a range of this size; without it, every request uses the same key |
| `-n` | Total requests per test |
| `-d` | Value size in bytes |
| `-t` | The tests to run (`set,get,incr,lpush,...`) |
| `--cluster` | Test a cluster |

`-t set,get` runs the `SET` test before the `GET` test, so the `GET` test
reads keys the `SET` test wrote. With `-r`, keys are chosen at random, so a
`SET` test with `-n` well above `-r` touches nearly every key, but not
necessarily all of them. For read tests where every lookup must hit, fill
the keyspace with memtier first, or run a long `SET` pass on its own.

## Testing your own workload

Synthetic `SET`/`GET` numbers rarely match an application. For sizing,
replay something closer to your traffic: your real value sizes, your mix of
commands (hashes, sorted sets, scripts), your TTLs, and your number of
client connections. memtier's `--command` option runs arbitrary commands
with key placeholders, and `--key-pattern=G:G` approximates a skewed access
pattern where some keys are much hotter than others.

## Next steps

- [Performance](https://docs.hotkv.com/performance/index.md) for HotKV's published results
- [Monitoring](https://docs.hotkv.com/operate/monitoring/index.md) to watch the server during a test
- [Configuration](https://docs.hotkv.com/operate/configuration/index.md)