Benchmarking

HotKV speaks the Redis protocol, so the standard tools work against it unchanged: redis-benchmark and memtier_benchmark. This page is about getting numbers you can trust, whether you are sizing a deployment or comparing HotKV with Redis or Valkey. HotKV’s own published results are on the Performance page.

What makes a benchmark fair

Most misleading benchmark results come from the setup, not the server. Before you compare numbers:

• Run the load generator on a different machine from the server, on the same network you will use in production. A client on the server’s host takes CPU from the server and skips the network entirely.
• Make sure the client is not the bottleneck. Give the load generator enough cores and threads, and watch its CPU during the run. If the client machine is at 100% CPU, you are measuring the client. Use several client machines if needed.
• Know your network ceiling. At high request rates or large values, the network link limits throughput before the server does. Check the bandwidth between the machines (for example with iperf3) and compare it with the bytes per second the benchmark moves.
• Use the same hardware and settings for every server you compare: same machine type, same persistence settings (all off, or all on with the same fsync policy), same maxmemory and eviction policy, same password or TLS settings.
• Give every server its threading. HotKV uses all CPU cores by default (threads 0). Redis and Valkey need io-threads set to use more than one core for network I/O. Configure each server the way you would run it in production, and write down what you used.
• Fill the keyspace before measuring reads. A GET on a key that does not exist returns nil without touching any data, so a read test against an empty or partly filled server measures misses. Load every key first, then check DBSIZE.
• Randomize key choice. Some tools use a fixed random seed by default, so every client requests the same sequence of keys. Use per-client random seeds so the access pattern looks like real traffic.
• Run long enough, and more than once. Discard a warm-up period, run for at least a minute, repeat each test a few times, and report the median. Report latency percentiles (p50, p99, p99.9) alongside throughput.
• State the pipeline depth. Results with pipelining and without are not comparable; report both if your application uses both.

Tip

Run CONFIG RESETSTAT on the server before each test. Afterwards, INFO commandstats shows the calls and server-side time per command, and INFO stats shows keyspace_hits and keyspace_misses: a read test should show hits, not misses.

memtier_benchmark

memtier_benchmark gives fine control over key patterns, ratios and client concurrency.

1. Fill the keyspace with every key the read test will use (here 1,000,000 keys of 64 bytes):

memtier_benchmark -s 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  --ratio=1:0 --key-pattern=P:P \
  --key-minimum=1 --key-maximum=1000000 \
  --data-size=64 --requests=allkeys \
  --threads=4 --clients=10 --hide-histogram

Check the result with redis-cli -h 10.0.0.1 -a "$HOTKV_PASSWORD" DBSIZE.

2. Run the test with random keys over the same range, a different random seed per client, and a fixed duration:

memtier_benchmark -s 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  --ratio=1:10 --key-pattern=R:R \
  --key-minimum=1 --key-maximum=1000000 \
  --data-size=64 --test-time=60 \
  --threads=8 --clients=50 --pipeline=1 \
  --randomize --distinct-client-seed

Option	What it controls
--threads, --clients	Load generator threads, and connections per thread. Raise them until throughput stops growing or latency climbs.
--pipeline	Requests in flight per connection. 1 is no pipelining.
--ratio=SET:GET	The write to read mix (1:0 writes only, 0:1 reads only)
--key-pattern	P:P walks the range in parallel blocks (for filling), R:R picks keys at random, G:G concentrates on the middle of the range
--randomize, --distinct-client-seed	A random seed for the run, and a different one per client
--data-size	Value size in bytes
--run-count	Repeat the test and report each run
--cluster-mode	Follow cluster redirects when testing a cluster

redis-benchmark

redis-benchmark is quick to run and covers many commands in one go:

redis-benchmark -h 10.0.0.1 -p 6379 -a "$HOTKV_PASSWORD" \
  -t set,get -n 10000000 -r 1000000 -d 64 \
  -c 200 --threads 8 -P 16 -q

Option	What it controls
-c	Total connections
--threads	Load generator threads; without it, one thread drives every connection and usually becomes the limit
-P	Pipeline depth (default 1, no pipelining)
-r	Use random keys from a range of this size; without it, every request uses the same key
-n	Total requests per test
-d	Value size in bytes
-t	The tests to run (set,get,incr,lpush,...)
--cluster	Test a cluster

-t set,get runs the SET test before the GET test, so the GET test reads keys the SET test wrote. With -r, keys are chosen at random, so a SET test with -n well above -r touches nearly every key, but not necessarily all of them. For read tests where every lookup must hit, fill the keyspace with memtier first, or run a long SET pass on its own.

Testing your own workload

Synthetic SET/GET numbers rarely match an application. For sizing, replay something closer to your traffic: your real value sizes, your mix of commands (hashes, sorted sets, scripts), your TTLs, and your number of client connections. memtier’s --command option runs arbitrary commands with key placeholders, and --key-pattern=G:G approximates a skewed access pattern where some keys are much hotter than others.

Next steps

• Performance for HotKV’s published results
• Monitoring to watch the server during a test
• Configuration