# Cluster on Kubernetes

> Run a six-pod HotKV cluster as a StatefulSet with a headless Service, create it with a Job, and watch it survive a deleted pod, a full restart and a rolling image update.

URL: https://docs.hotkv.com/clustering/kubernetes/


**Availability:** Not on HotKV Cloud. On-Premise: no license needed.

A HotKV cluster fits a Kubernetes StatefulSet well: every node needs a stable
name, its own volume and a way to be found by the others when it comes back
with a new address, and a StatefulSet with a headless Service gives it all
three. This page applies one manifest that creates six pods (three primaries
and three replicas), a Secret for the password, a volume for each pod and a
Job that forms the cluster. Then it deletes a pod, deletes every pod, and
updates the image while a client keeps writing.

This is a starting point to adapt.

## What you need

- A Kubernetes cluster and `kubectl`. The examples use
  [kind](https://kind.sigs.k8s.io/) with its default storage class, one node.
- The HotKV image, called `hotkv` here: use the name and tag you were given,
  and make it available to the cluster. With kind that is
  `kind load docker-image hotkv`; with a real cluster, push it to a registry
  the cluster can pull from and put the full name in the manifest. The manifest
  sets `imagePullPolicy: IfNotPresent` so that an image loaded into kind is
  used.
- A storage class that gives each pod a volume (the `volumeClaimTemplates`
  use the default one).

## The manifest

Save this as `hotkv-cluster.yaml`. Change the password in the Secret first:

```yaml title="hotkv-cluster.yaml"
apiVersion: v1
kind: Secret
metadata:
  name: hotkv
type: Opaque
stringData:
  password: change-this-to-a-long-random-password
---
apiVersion: v1
kind: Service
metadata:
  name: hotkv
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: hotkv
  ports:
    - name: client
      port: 6379
    - name: bus
      port: 16379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hotkv
spec:
  serviceName: hotkv
  replicas: 6
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: hotkv
  template:
    metadata:
      labels:
        app: hotkv
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: hotkv
          image: hotkv
          imagePullPolicy: IfNotPresent
          args:
            - --cluster-enabled
            - "yes"
            - --requirepass
            - $(HOTKV_PASSWORD)
            - --masterauth
            - $(HOTKV_PASSWORD)
            - --appendonly
            - "yes"
            - --cluster-announce-ip
            - $(POD_NAME).hotkv.$(POD_NAMESPACE).svc
            - --shutdown-on-sigterm
            - failover
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: HOTKV_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
            - name: HOTKVCLI_AUTH
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
          ports:
            - name: client
              containerPort: 6379
            - name: bus
              containerPort: 16379
          readinessProbe:
            exec:
              command: ["hotkv-cli", "ping"]
            periodSeconds: 5
          livenessProbe:
            tcpSocket:
              port: 6379
            periodSeconds: 10
            failureThreshold: 6
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hotkv-init
data:
  init.sh: |
    set -eu
    cli() { hotkv-cli -h "hotkv-$1.hotkv" "${@:2}"; }

    for i in 0 1 2 3 4 5; do
      until cli $i ping >/dev/null 2>&1; do sleep 1; done
    done

    if cli 0 cluster info | grep -q 'cluster_state:ok'; then
      echo "The cluster already exists."
      exit 0
    fi

    # Introduce every pod to pod 0, which tells the others.
    for i in 1 2 3 4 5; do
      cli 0 cluster meet "$(getent hosts "hotkv-$i.hotkv" | cut -d' ' -f1)" 6379
    done
    until [ "$(cli 0 cluster nodes | grep -c .)" = 6 ]; do sleep 1; done

    # Pods 0, 1 and 2 are the primaries and share the 16384 slots.
    cli 0 cluster addslotsrange 0 5460
    cli 1 cluster addslotsrange 5461 10922
    cli 2 cluster addslotsrange 10923 16383

    # Pods 3, 4 and 5 become their replicas.
    for pair in 3:0 4:1 5:2; do
      replica=${pair%%:*}; primary=${pair##*:}
      until cli "$replica" cluster nodes | grep -q "$(cli "$primary" cluster myid)"; do sleep 1; done
      cli "$replica" cluster replicate "$(cli "$primary" cluster myid)"
    done

    for i in 0 1 2 3 4 5; do
      until cli $i cluster info | grep -q 'cluster_state:ok'; do sleep 1; done
    done
    echo "The cluster is ready."
---
apiVersion: batch/v1
kind: Job
metadata:
  name: hotkv-init
spec:
  backoffLimit: 6
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: init
          image: hotkv
          imagePullPolicy: IfNotPresent
          command: ["/bin/bash", "/scripts/init.sh"]
          env:
            - name: HOTKVCLI_AUTH
              valueFrom:
                secretKeyRef:
                  name: hotkv
                  key: password
          volumeMounts:
            - name: scripts
              mountPath: /scripts
      volumes:
        - name: scripts
          configMap:
            name: hotkv-init
```

The settings that matter for a cluster:

- **The headless Service** (`clusterIP: None`) gives each pod a DNS name,
  `hotkv-0.hotkv.<namespace>.svc`. `publishNotReadyAddresses: true` publishes
  the names as soon as the pods exist, whether or not they are ready yet.
- **`--cluster-announce-ip $(POD_NAME).hotkv.$(POD_NAMESPACE).svc`** makes each
  pod announce its DNS name instead of its address. Pods get a new address when
  they are rescheduled, and the name keeps pointing at them. Announcing the
  pod address (`status.podIP`) is not recommended: after every pod has been
  restarted at once (a node drain, a power loss) the nodes remember addresses
  that no longer exist, and the cluster may not recover. An announced address must be shorter than 46 characters;
  `hotkv-0.hotkv.default.svc` has 25, so a namespace name of up to 27
  characters fits.
- **`--requirepass` and `--masterauth`** use the same password from the Secret.
  Replicas and an atomic slot migration use it to log in to a primary.
- **`--shutdown-on-sigterm failover`**: when Kubernetes stops a primary pod it
  sends SIGTERM, and with this setting the primary hands its slots to a replica
  before it exits. A deleted pod, a node drain and a rolling update then cause
  no outage.
- **`terminationGracePeriodSeconds: 120`** gives a pod time to hand over and
  save its data.
- **The readiness probe** is `hotkv-cli ping` (the image has the password in
  `HOTKVCLI_AUTH`, which the manifest sets). It does not depend on the cluster
  state, so pods are ready before the cluster exists and the Job can reach
  them. The liveness probe only checks that the port answers.
- **`podManagementPolicy: Parallel`** starts all six pods at once. The Job
  creates the cluster, so the order does not matter.
- **The init Job** is the same script as in
  [Cluster in Docker Compose](https://docs.hotkv.com/clustering/docker-compose/index.md), with
  `hotkv-N.hotkv` as the node names. It is safe to run again: it exits when the
  cluster already exists.

## Create the cluster

```bash
kubectl apply -f hotkv-cluster.yaml
kubectl wait --for=condition=complete job/hotkv-init --timeout=180s
kubectl logs job/hotkv-init | tail -3
```

```text
secret/hotkv created
service/hotkv created
statefulset.apps/hotkv created
configmap/hotkv-init created
job.batch/hotkv-init created
job.batch/hotkv-init condition met
OK
OK
The cluster is ready.
```

```bash
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,IP:.status.podIP
kubectl exec hotkv-0 -- hotkv-cli cluster info | head -3
kubectl exec hotkv-0 -- hotkv-cli cluster nodes | awk '{print $2, $3, $9}' | sort
```

```text
NAME      READY   IP
hotkv-0   true    10.244.0.122
hotkv-1   true    10.244.0.124
hotkv-2   true    10.244.0.125
hotkv-3   true    10.244.0.127
hotkv-4   true    10.244.0.128
hotkv-5   true    10.244.0.129
cluster_enabled:1
cluster_state:ok
cluster_slots_assigned:16384
hotkv-0.hotkv.default.svc:6379@16379 myself,master 0-5460
hotkv-1.hotkv.default.svc:6379@16379 master 5461-10922
hotkv-2.hotkv.default.svc:6379@16379 master 10923-16383
hotkv-3.hotkv.default.svc:6379@16379 slave
hotkv-4.hotkv.default.svc:6379@16379 slave
hotkv-5.hotkv.default.svc:6379@16379 slave
```

Pods 0, 1 and 2 are the primaries and 3, 4 and 5 their replicas. `hotkv-cli`
runs inside the pod and picks up the password from the environment; it does
not follow redirects, so use it to look at one node. The nodes announce their
DNS names, so `CLUSTER NODES` and every `MOVED` carry them.

## Connect an application

Applications in the same cluster use any pod as a seed node and a
cluster-aware client library (see [Cluster clients](https://docs.hotkv.com/clustering/clients/index.md)).
The seed is `hotkv-0.hotkv:6379` from the same namespace, or
`hotkv-0.hotkv.<namespace>.svc:6379` from another one, with the password from
the Secret. The load generator in the last section shows one in a pod.

The names resolve only inside the Kubernetes cluster. Reaching the cluster from
outside needs every node announced at an address the outside client can reach
(see [Deploy on VMs and cloud](https://docs.hotkv.com/clustering/production/index.md)); that is not covered
here.

## Delete a pod

Write a key to the primary of its slot, then delete that pod. Its replica takes
over, and the pod that comes back, with a new IP address, rejoins as a replica:

```bash
kubectl exec hotkv-0 -- hotkv-cli set user:1000 ada
kubectl get pod hotkv-0 -o jsonpath="{.status.podIP}{\"\n\"}"
kubectl delete pod hotkv-0
kubectl wait --for=condition=Ready pod/hotkv-0 --timeout=120s
kubectl get pod hotkv-0 -o jsonpath="{.status.podIP}{\"\n\"}"
kubectl exec hotkv-3 -- hotkv-cli get user:1000
kubectl exec hotkv-0 -- hotkv-cli cluster nodes | awk '{print $2, $3, $9}' | sort
```

```text
OK
10.244.0.122
pod "hotkv-0" deleted from default namespace
pod/hotkv-0 condition met
10.244.0.130
ada
hotkv-0.hotkv.default.svc:6379@16379 myself,slave
hotkv-1.hotkv.default.svc:6379@16379 master 5461-10922
hotkv-2.hotkv.default.svc:6379@16379 master 10923-16383
hotkv-3.hotkv.default.svc:6379@16379 master 0-5460
hotkv-4.hotkv.default.svc:6379@16379 slave
hotkv-5.hotkv.default.svc:6379@16379 slave
```

`hotkv-3`, the replica of `hotkv-0`, now serves slots 0 to 5460 and has the
key, and `hotkv-0` is its replica with a new address.

## Delete every pod

A node drain or a power loss restarts all the pods at once, and every pod
gets a new address. Because the pods announce names, the cluster forms
again by itself, with its data (the volumes keep each pod's data and
`nodes.conf`):

```bash
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IP:.status.podIP | head -3
kubectl delete pod -l app=hotkv > /dev/null; kubectl wait --for=condition=Ready pod -l app=hotkv --timeout=180s | tail -1
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IP:.status.podIP | head -3
until kubectl exec hotkv-3 -- hotkv-cli cluster info | grep -q cluster_state:ok; do sleep 2; done
kubectl exec hotkv-1 -- hotkv-cli cluster info | head -2
kubectl exec hotkv-3 -- hotkv-cli get user:1000
```

```text
NAME      IP
hotkv-0   10.244.0.130
hotkv-1   10.244.0.124
pod/hotkv-5 condition met
NAME      IP
hotkv-0   10.244.0.134
hotkv-1   10.244.0.132
cluster_enabled:1
cluster_state:ok
ada
```

(The cluster takes a few seconds to agree after the pods are ready, which is
what the `until` loop waits for. Without it, `hotkv-3` can still answer
`CLUSTERDOWN The cluster is down` right after the pods are ready.)

## Update the image with a client running

A StatefulSet updates its pods one at a time, from the highest ordinal to the
lowest, and waits for each to be ready. With `--shutdown-on-sigterm failover`
every primary pod hands over to its replica before it is replaced. To see what
a client notices, run a small load generator in the cluster. It is a Python
program using the redis-py cluster client that writes and reads back a key
every 20 milliseconds for 130 seconds and counts failures:

```yaml title="loadgen-pod.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
  name: loadgen
data:
  loadgen.py: |
    import os, time
    from redis.cluster import RedisCluster

    rc = RedisCluster(host="hotkv-0.hotkv", port=6379, password=os.environ["HOTKV_PASSWORD"],
                      decode_responses=True)
    duration = int(os.environ.get("DURATION", "120"))
    ok = errors = 0
    last_error = None
    start = time.time()
    n = 0
    while time.time() - start < duration:
        key = f"user:{n % 500}"
        try:
            rc.set(key, n)
            rc.get(key)
            ok += 1
        except Exception as exc:
            errors += 1
            last_error = f"{type(exc).__name__}: {exc}"
        n += 1
        time.sleep(0.02)
    print(f"operations ok: {ok}, failed: {errors}")
    if errors:
        print(f"last error: {last_error}")
---
apiVersion: v1
kind: Pod
metadata:
  name: loadgen
spec:
  restartPolicy: Never
  containers:
    - name: loadgen
      image: python:3-slim
      command: ["sh", "-c", "pip install --quiet --root-user-action=ignore redis && python /app/loadgen.py"]
      env:
        - name: DURATION
          value: "130"
        - name: HOTKV_PASSWORD
          valueFrom:
            secretKeyRef:
              name: hotkv
              key: password
      volumeMounts:
        - name: app
          mountPath: /app
  volumes:
    - name: app
      configMap:
        name: loadgen
```

Start it, then change the image tag of the StatefulSet (here `hotkv:v2`) and wait for the
roll-out:

```bash
kubectl apply -f loadgen-pod.yaml
kubectl set image statefulset/hotkv hotkv=hotkv:v2
kubectl rollout status statefulset/hotkv --timeout=300s | tail -2
kubectl wait --for=jsonpath="{.status.phase}"=Succeeded pod/loadgen --timeout=200s
kubectl logs loadgen
kubectl get pods -l app=hotkv -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
kubectl exec hotkv-0 -- hotkv-cli cluster info | head -2
```

```text
configmap/loadgen created
pod/loadgen created
statefulset.apps/hotkv image updated
Waiting for 1 pods to be ready...
partitioned roll out complete: 6 new pods have been updated...
pod/loadgen condition met
operations ok: <n>, failed: 0
NAME      IMAGE
hotkv-0   hotkv:v2
hotkv-1   hotkv:v2
hotkv-2   hotkv:v2
hotkv-3   hotkv:v2
hotkv-4   hotkv:v2
hotkv-5   hotkv:v2
cluster_enabled:1
cluster_state:ok
```

Start the load generator before you change the image, so that it runs while the
pods are replaced.

A pod that is killed instead of stopped, or a node that disappears, gives the
primary no chance to hand over. Its replica takes over only after the other
nodes agree the primary is gone, which takes the node timeout (15 seconds by
default) plus the election (see the kill example on the
[Compose page](https://docs.hotkv.com/clustering/docker-compose/index.md#fail-a-primary)).

## Clean up

```bash
kubectl delete -f hotkv-cluster.yaml -f loadgen-pod.yaml
kubectl delete pvc -l app=hotkv
```

Deleting the StatefulSet does not delete its volumes. The second command does,
and with them the cluster's data.

## What is not covered

- **Adding or removing nodes**, which is a
  [scaling](https://docs.hotkv.com/clustering/scaling/index.md) operation you run yourself.
- **Access from outside the Kubernetes cluster**, and TLS between the nodes
  ([Cluster TLS](https://docs.hotkv.com/clustering/tls/index.md)).
- **Resource requests and limits, anti-affinity and pod disruption budgets**
  for a multi-node Kubernetes cluster.
- **Licenses.** Cluster mode needs none; for the AI engines every pod needs its
  license (see [AI engines in a cluster](https://docs.hotkv.com/clustering/ai-engines/index.md)).

## Next steps

- [Cluster in Docker Compose](https://docs.hotkv.com/clustering/docker-compose/index.md): the same layout on
  one machine.
- [Cluster clients](https://docs.hotkv.com/clustering/clients/index.md): retries through failover and
  reshard.
- [Docker and Kubernetes](https://docs.hotkv.com/operate/docker-and-kubernetes/index.md): the single-node
  image and what the image does.