> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo A Rolling Update Under Live Traffic

> Demonstrates zero-downtime rolling updates for model servers under live traffic using a user heartbeat, Kubernetes rollout, and Prometheus readiness monitoring

Time to prove the workhorse: update the model-server fleet while requests are flowing, with the success condition being simple — nobody notices.

This demo uses three perspectives:

* A user terminal generating a steady heartbeat of requests.
* An operator terminal performing the rollout.
* Prometheus as a witness that watches readiness and capacity.

## 1) Users: a steady heartbeat of requests

On the user terminal we run a simple loop that sends one request per second and prints the request sequence number and HTTP status. A continuous column of `200`s is the heartbeat; any non-`200` shows up immediately.

```bash theme={null}
i=0
while true; do
  i=$((i+1))
  curl -s -o /dev/null -w "request ${i}: %{http_code}\n" \
    http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"prompt\":\"Say one word about update number ${i}.\",\"max_tokens\":5}"
  sleep 1
done
```

Leave this running in the background on the first terminal. It will continuously print lines like:

```console theme={null}
request 1: 200
request 2: 200
request 3: 200
...
```

This simple client verifies availability from the callers’ perspective.

## 2) Operator: rolling the deployment

On the operator terminal we locate the `decode-role` deployment in the namespace, trigger a rollout (here simulated by setting an environment variable), and then watch pods as Kubernetes performs the rolling update.

```bash theme={null}
export NAMESPACE=llm-d

DECODE_DEPLOY=$(kubectl get deploy -n ${NAMESPACE} -l llm-d.ai/role=decode \
  -o jsonpath='{.items[0].metadata.name}')

kubectl set env deploy/${DECODE_DEPLOY} -n ${NAMESPACE} DEMO_ROLLOUT=v42
kubectl get pods -n ${NAMESPACE} -w
```

Setting or changing an environment variable is a safe stand‑in for a real trigger (for example, a new image tag or server version). Kubernetes treats this as a pod spec change and performs the same rolling replacement sequence.

While watching pods you’ll observe the rollout lifecycle. A representative snapshot looks like this:

```console theme={null}
deployment.apps/optimized-baseline-cpu-vllm-decode env updated
NAME                                                   READY   STATUS            RESTARTS   AGE
optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb    1/1     Running           0          5d22h
optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd    1/1     Running           0          5d22h
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb    1/1     Terminating       0          5d22h
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     Pending           0          2s
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     ContainerCreating 0          3s
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     Running           0          3s
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb    0/1     Completed         0          5d22h
```

Watch the lifecycle carefully:

* A new pod appears and typically moves Pending → ContainerCreating → Running, but its readiness probe will initially report “not ready”.
* While not Ready the pod does not receive traffic — it may be pulling model weights and initializing.
* Once the readiness probe passes, the pod flips to Ready and joins the inference pool.
* Only after the replacement is Ready does an old pod begin Terminating, drain in‑flight work, and exit.

This one‑by‑one turnover preserves capacity throughout the update.

## 3) Prometheus: verifying ready endpoints

From Prometheus we can inspect what the router sees for ready endpoints. The metric below shows how many ready endpoints the router sees for the `quickstart-epp` service:

```text theme={null}
llm_d_epp_ready_endpoints
llm_d_epp_ready_endpoints{container="epp", endpoint="http-metrics", instance="10.244.223.194:9090", job="quickstart-epp", name="quickstart", namespace="llm-d", pod="quickstart-epp-5f6cbf8c4-tcl7q", service="quickstart-epp"} 2
```

That count may wobble during the handover but — crucially — it never reaches `0`. The rollout strategy ensures the system never reduces available capacity before a replacement is ready.

## Observing the user stream during turnover

Back on the user terminal you should continue to see requests succeed:

```console theme={null}
request 101: 200
request 102: 200
request 103: 200
request 104: 200
...
```

No gaps, no error fingerprints — the update completed transparently from the callers’ perspective.

## Why this is safe: three cooperating mechanisms

| Mechanism | Role in zero-downtime updates | Notes / links |
| - | - | - |
| Rolling update strategy | Replaces pods one at a time so total capacity does not drop below the configured target | See Kubernetes rolling update docs: [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment) |
| Readiness gating | New pods are invisible to the router until the model is fully initialized and can answer requests | Readiness probes: [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) |
| Router tracking | Router tracks pool membership live and only sends traffic to Ready endpoints | Router consults readiness-aware metrics (e.g., from Prometheus) |

Together these ensure an update happens with no observable interruption: pods ready only when healthy, router only sends traffic to ready endpoints, and the rolling strategy prevents capacity dips. That’s day‑two operations in miniature — queues hold the line, autoscaling and readiness signals are trusted, and change leaves no fingerprints.

## Quick command reference

| Purpose | Command |
| -: | - |
| Stream test requests | See the user loop above |
| Find decode deployment | `kubectl get deploy -n llm-d -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}'` |
| Trigger rollout (env var) | `kubectl set env deploy/<name> -n llm-d DEMO_ROLLOUT=v42` |
| Watch pods | `kubectl get pods -n llm-d -w` |

## Links and references

* Kubernetes rolling updates: [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment)
* Readiness probes: [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
* Prometheus: [https://prometheus.io/](https://prometheus.io/)

A quick tip: run the user heartbeat and the operator rollout in separate terminals to see the real‑time effect. Use Prometheus or your router’s metrics to validate that ready endpoint counts never drop to zero.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/ab1e251e-2849-4546-a794-c83a0f053ce7/lesson/9c54bb62-3283-4691-8c75-cff2ffe2fc51)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.