Skip to content
HotKVDocs

Ask the HotKV docs

Answers come only from these docs and can be wrong; check the linked pages before you rely on them.

Cluster on Kubernetes

Last updated
Not on HotKV CloudOn-Premise: no license needed

A HotKV cluster fits a Kubernetes StatefulSet well: every node needs a stable name, its own volume and a way to be found by the others when it comes back with a new address, and a StatefulSet with a headless Service gives it all three. This page applies one manifest that creates six pods (three primaries and three replicas), a Secret for the password, a volume for each pod and a Job that forms the cluster. Then it deletes a pod, deletes every pod, and updates the image while a client keeps writing.

This is a starting point to adapt.

What you need

Section titled “What you need”
  • A Kubernetes cluster and kubectl. The examples use kind with its default storage class, one node.
  • The HotKV image, called hotkv here: use the name and tag you were given, and make it available to the cluster. With kind that is kind load docker-image hotkv; with a real cluster, push it to a registry the cluster can pull from and put the full name in the manifest. The manifest sets imagePullPolicy: IfNotPresent so that an image loaded into kind is used.
  • A storage class that gives each pod a volume (the volumeClaimTemplates use the default one).

The manifest

Section titled “The manifest”

Save this as hotkv-cluster.yaml. Change the password in the Secret first:

hotkv-cluster.yaml
apiVersion: v1
kind: Secret
metadata:
  name: hotkv
type: Opaque
stringData:
  password: change-this-to-a-long-random-password
---
apiVersion: v1
kind: Service
metadata:
  name: hotkv
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: hotkv
  ports:
    - name: client
      port: 6379
    - name: bus
      port: 16379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hotkv
spec:
  serviceName: hotkv
  replicas: 6
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: hotkv
  template:
    metadata:
      labels:
        app: hotkv
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: hotkv
          image: hotkv
          imagePullPolicy: IfNotPresent
          args:
            - --cluster-enabled
            - "yes"
            - --requirepass
            - $(HOTKV_PASSWORD)
            - --masterauth
            - $(HOTKV_PASSWORD)
            - --appendonly
            - "yes"
            - --cluster-announce-ip
            - $(POD_NAME).hotkv.$(POD_NAMESPACE).svc
            - --shutdown-on-sigterm
            - failover
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: HOTKV_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
            - name: HOTKVCLI_AUTH
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
          ports:
            - name: client
              containerPort: 6379
            - name: bus
              containerPort: 16379
          readinessProbe:
            exec:
              command: ["hotkv-cli", "ping"]
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 6379
            periodSeconds: 10
            failureThreshold: 6
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hotkv-init
data:
  init.sh: |
    set -eu
    cli() { hotkv-cli -h "hotkv-$1.hotkv" "${@:2}"; }


    for i in 0 1 2 3 4 5; do
      until cli $i ping >/dev/null 2>&1; do sleep 1; done
    done


    if cli 0 cluster info | grep -q 'cluster_state:ok'; then
      echo "The cluster already exists."
      exit 0
    fi


    # Introduce every pod to pod 0, which tells the others.
    for i in 1 2 3 4 5; do
      cli 0 cluster meet "$(getent hosts "hotkv-$i.hotkv" | cut -d' ' -f1)" 6379
    done
    until [ "$(cli 0 cluster nodes | grep -c .)" = 6 ]; do sleep 1; done


    # Pods 0, 1 and 2 are the primaries and share the 16384 slots.
    cli 0 cluster addslotsrange 0 5460
    cli 1 cluster addslotsrange 5461 10922
    cli 2 cluster addslotsrange 10923 16383


    # Pods 3, 4 and 5 become their replicas.
    for pair in 3:0 4:1 5:2; do
      replica=${pair%%:*}; primary=${pair##*:}
      until cli "$replica" cluster nodes | grep -q "$(cli "$primary" cluster myid)"; do sleep 1; done
      cli "$replica" cluster replicate "$(cli "$primary" cluster myid)"
    done


    for i in 0 1 2 3 4 5; do
      until cli $i cluster info | grep -q 'cluster_state:ok'; do sleep 1; done
    done
    echo "The cluster is ready."
---
apiVersion: batch/v1
kind: Job
metadata:
  name: hotkv-init
spec:
  backoffLimit: 6
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: init
          image: hotkv
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "/scripts/init.sh"]
          env:
            - name: HOTKVCLI_AUTH
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: hotkv-init

The settings that matter for a cluster:

  • The headless Service (clusterIP: None) gives each pod a DNS name, hotkv-0.hotkv.<namespace>.svc. publishNotReadyAddresses: true publishes the names as soon as the pods exist, whether or not they are ready yet.
  • --cluster-announce-ip $(POD_NAME).hotkv.$(POD_NAMESPACE).svc makes each pod announce its DNS name instead of its address. Pods get a new address when they are rescheduled, and the name keeps pointing at them. Announcing the pod address (status.podIP) is not recommended: after every pod has been restarted at once (a node drain, a power loss) the nodes remember addresses that no longer exist, and the cluster may not recover. An announced address must be shorter than 46 characters; hotkv-0.hotkv.default.svc has 25, so a namespace name of up to 27 characters fits.
  • --requirepass and --masterauth use the same password from the Secret. Replicas and an atomic slot migration use it to log in to a primary.
  • --shutdown-on-sigterm failover: when Kubernetes stops a primary pod it sends SIGTERM, and with this setting the primary hands its slots to a replica before it exits. A deleted pod, a node drain and a rolling update then cause no outage.
  • terminationGracePeriodSeconds: 120 gives a pod time to hand over and save its data.
  • The readiness probe is hotkv-cli ping (the image has the password in HOTKVCLI_AUTH, which the manifest sets). It does not depend on the cluster state, so pods are ready before the cluster exists and the Job can reach them. The liveness probe only checks that the port answers.
  • podManagementPolicy: Parallel starts all six pods at once. The Job creates the cluster, so the order does not matter.
  • The init Job is the same script as in Cluster in Docker Compose, with hotkv-N.hotkv as the node names. It is safe to run again: it exits when the cluster already exists.

Create the cluster

Section titled “Create the cluster”
Terminal window
kubectl apply -f hotkv-cluster.yaml
kubectl wait --for=condition=complete job/hotkv-init --timeout=180s
kubectl logs job/hotkv-init | tail -3
secret/hotkv created
service/hotkv created
statefulset.apps/hotkv created
configmap/hotkv-init created
job.batch/hotkv-init created
job.batch/hotkv-init condition met
OK
OK
The cluster is ready.
Terminal window
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,IP:.status.podIP
kubectl exec hotkv-0 -- hotkv-cli cluster info | head -3
kubectl exec hotkv-0 -- hotkv-cli cluster nodes | awk '{print $2, $3, $9}' | sort
NAME      READY   IP
hotkv-0   true    10.244.0.122
hotkv-1   true    10.244.0.124
hotkv-2   true    10.244.0.125
hotkv-3   true    10.244.0.127
hotkv-4   true    10.244.0.128
hotkv-5   true    10.244.0.129
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
hotkv-0.hotkv.default.svc:6379@16379 myself,master 0-5460
hotkv-1.hotkv.default.svc:6379@16379 master 5461-10922
hotkv-2.hotkv.default.svc:6379@16379 master 10923-16383
hotkv-3.hotkv.default.svc:6379@16379 slave
hotkv-4.hotkv.default.svc:6379@16379 slave
hotkv-5.hotkv.default.svc:6379@16379 slave

Pods 0, 1 and 2 are the primaries and 3, 4 and 5 their replicas. hotkv-cli runs inside the pod and picks up the password from the environment; it does not follow redirects, so use it to look at one node. The nodes announce their DNS names, so CLUSTER NODES and every MOVED carry them.

Connect an application

Section titled “Connect an application”

Applications in the same cluster use any pod as a seed node and a cluster-aware client library (see Cluster clients). The seed is hotkv-0.hotkv:6379 from the same namespace, or hotkv-0.hotkv.<namespace>.svc:6379 from another one, with the password from the Secret. The load generator in the last section shows one in a pod.

The names resolve only inside the Kubernetes cluster. Reaching the cluster from outside needs every node announced at an address the outside client can reach (see Deploy on VMs and cloud); that is not covered here.

Delete a pod

Section titled “Delete a pod”

Write a key to the primary of its slot, then delete that pod. Its replica takes over, and the pod that comes back, with a new IP address, rejoins as a replica:

Terminal window
kubectl exec hotkv-0 -- hotkv-cli set user:1000 ada
kubectl get pod hotkv-0 -o jsonpath="{.status.podIP}{\"\n\"}"
kubectl delete pod hotkv-0
kubectl wait --for=condition=Ready pod/hotkv-0 --timeout=120s
kubectl get pod hotkv-0 -o jsonpath="{.status.podIP}{\"\n\"}"
kubectl exec hotkv-3 -- hotkv-cli get user:1000
kubectl exec hotkv-0 -- hotkv-cli cluster nodes | awk '{print $2, $3, $9}' | sort
OK
10.244.0.122
pod "hotkv-0" deleted from default namespace
pod/hotkv-0 condition met
10.244.0.130
ada
hotkv-0.hotkv.default.svc:6379@16379 myself,slave
hotkv-1.hotkv.default.svc:6379@16379 master 5461-10922
hotkv-2.hotkv.default.svc:6379@16379 master 10923-16383
hotkv-3.hotkv.default.svc:6379@16379 master 0-5460
hotkv-4.hotkv.default.svc:6379@16379 slave
hotkv-5.hotkv.default.svc:6379@16379 slave

hotkv-3, the replica of hotkv-0, now serves slots 0 to 5460 and has the key, and hotkv-0 is its replica with a new address.

Delete every pod

Section titled “Delete every pod”

A node drain or a power loss restarts all the pods at once, and every pod gets a new address. Because the pods announce names, the cluster forms again by itself, with its data (the volumes keep each pod’s data and nodes.conf):

Terminal window
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IP:.status.podIP | head -3
kubectl delete pod -l app=hotkv > /dev/null; kubectl wait --for=condition=Ready pod -l app=hotkv --timeout=180s | tail -1
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IP:.status.podIP | head -3
until kubectl exec hotkv-3 -- hotkv-cli cluster info | grep -q cluster_state:ok; do sleep 2; done
kubectl exec hotkv-1 -- hotkv-cli cluster info | head -2
kubectl exec hotkv-3 -- hotkv-cli get user:1000
NAME      IP
hotkv-0   10.244.0.130
hotkv-1   10.244.0.124
pod/hotkv-5 condition met
NAME      IP
hotkv-0   10.244.0.134
hotkv-1   10.244.0.132
cluster_enabled:1
cluster_state:ok
ada

(The cluster takes a few seconds to agree after the pods are ready, which is what the until loop waits for. Without it, hotkv-3 can still answer CLUSTERDOWN The cluster is down right after the pods are ready.)

Update the image with a client running

Section titled “Update the image with a client running”

A StatefulSet updates its pods one at a time, from the highest ordinal to the lowest, and waits for each to be ready. With --shutdown-on-sigterm failover every primary pod hands over to its replica before it is replaced. To see what a client notices, run a small load generator in the cluster. It is a Python program using the redis-py cluster client that writes and reads back a key every 20 milliseconds for 130 seconds and counts failures:

loadgen-pod.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: loadgen
data:
  loadgen.py: |
    import os, time
    from redis.cluster import RedisCluster


    rc = RedisCluster(host="hotkv-0.hotkv", port=6379, password=os.environ["HOTKV_PASSWORD"],
                      decode_responses=True)
    duration = int(os.environ.get("DURATION", "120"))
    ok = errors = 0
    last_error = None
    start = time.time()
    n = 0
    while time.time() - start < duration:
        key = f"user:{n % 500}"
        try:
            rc.set(key, n)
            rc.get(key)
            ok += 1
        except Exception as exc:
            errors += 1
            last_error = f"{type(exc).__name__}: {exc}"
        n += 1
        time.sleep(0.02)
    print(f"operations ok: {ok}, failed: {errors}")
    if errors:
        print(f"last error: {last_error}")
---
apiVersion: v1
kind: Pod
metadata:
  name: loadgen
spec:
  restartPolicy: Never
  containers:
    - name: loadgen
      image: python:3-slim
      command: ["sh", "-c", "pip install --quiet --root-user-action=ignore redis && python /app/loadgen.py"]
      env:
        - name: DURATION
          value: "130"
        - name: HOTKV_PASSWORD
          valueFrom:
            secretKeyRef:
              name: hotkv
              key: password
      volumeMounts:
        - name: app
          mountPath: /app
  volumes:
    - name: app
      configMap:
        name: loadgen

Start it, then change the image tag of the StatefulSet (here hotkv:v2) and wait for the roll-out:

Terminal window
kubectl apply -f loadgen-pod.yaml
kubectl set image statefulset/hotkv hotkv=hotkv:v2
kubectl rollout status statefulset/hotkv --timeout=300s | tail -2
kubectl wait --for=jsonpath="{.status.phase}"=Succeeded pod/loadgen --timeout=200s
kubectl logs loadgen
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
kubectl exec hotkv-0 -- hotkv-cli cluster info | head -2
configmap/loadgen created
pod/loadgen created
statefulset.apps/hotkv image updated
Waiting for 1 pods to be ready...
partitioned roll out complete: 6 new pods have been updated...
pod/loadgen condition met
operations ok: <n>, failed: 0
NAME      IMAGE
hotkv-0   hotkv:v2
hotkv-1   hotkv:v2
hotkv-2   hotkv:v2
hotkv-3   hotkv:v2
hotkv-4   hotkv:v2
hotkv-5   hotkv:v2
cluster_enabled:1
cluster_state:ok

Start the load generator before you change the image, so that it runs while the pods are replaced.

A pod that is killed instead of stopped, or a node that disappears, gives the primary no chance to hand over. Its replica takes over only after the other nodes agree the primary is gone, which takes the node timeout (15 seconds by default) plus the election (see the kill example on the Compose page).

Clean up

Section titled “Clean up”
Terminal window
kubectl delete -f hotkv-cluster.yaml -f loadgen-pod.yaml
kubectl delete pvc -l app=hotkv

Deleting the StatefulSet does not delete its volumes. The second command does, and with them the cluster’s data.

What is not covered

Section titled “What is not covered”
  • Adding or removing nodes, which is a scaling operation you run yourself.
  • Access from outside the Kubernetes cluster, and TLS between the nodes (Cluster TLS).
  • Resource requests and limits, anti-affinity and pod disruption budgets for a multi-node Kubernetes cluster.
  • Licenses. Cluster mode needs none; for the AI engines every pod needs its license (see AI engines in a cluster).

Next steps

Section titled “Next steps”