Demo A Rolling Update Under Live Traffic

Time to prove the workhorse: update the model-server fleet while requests are flowing, with the success condition being simple — nobody notices. This demo uses three perspectives:

A user terminal generating a steady heartbeat of requests.
An operator terminal performing the rollout.
Prometheus as a witness that watches readiness and capacity.

1) Users: a steady heartbeat of requests

On the user terminal we run a simple loop that sends one request per second and prints the request sequence number and HTTP status. A continuous column of 200s is the heartbeat; any non-200 shows up immediately.

i=0
while true; do
  i=$((i+1))
  curl -s -o /dev/null -w "request ${i}: %{http_code}\n" \
    http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"prompt\":\"Say one word about update number ${i}.\",\"max_tokens\":5}"
  sleep 1
done

Leave this running in the background on the first terminal. It will continuously print lines like:

request 1: 200
request 2: 200
request 3: 200
...

This simple client verifies availability from the callers’ perspective.

2) Operator: rolling the deployment

On the operator terminal we locate the decode-role deployment in the namespace, trigger a rollout (here simulated by setting an environment variable), and then watch pods as Kubernetes performs the rolling update.

export NAMESPACE=llm-d

DECODE_DEPLOY=$(kubectl get deploy -n ${NAMESPACE} -l llm-d.ai/role=decode \
  -o jsonpath='{.items[0].metadata.name}')

kubectl set env deploy/${DECODE_DEPLOY} -n ${NAMESPACE} DEMO_ROLLOUT=v42
kubectl get pods -n ${NAMESPACE} -w

Setting or changing an environment variable is a safe stand‑in for a real trigger (for example, a new image tag or server version). Kubernetes treats this as a pod spec change and performs the same rolling replacement sequence.

While watching pods you’ll observe the rollout lifecycle. A representative snapshot looks like this:

deployment.apps/optimized-baseline-cpu-vllm-decode env updated
NAME                                                   READY   STATUS            RESTARTS   AGE
optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb    1/1     Running           0          5d22h
optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd    1/1     Running           0          5d22h
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb    1/1     Terminating       0          5d22h
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     Pending           0          2s
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     ContainerCreating 0          3s
optimized-baseline-cpu-vllm-decode-7b976466db-w26xt    0/1     Running           0          3s
optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb    0/1     Completed         0          5d22h

Watch the lifecycle carefully:

A new pod appears and typically moves Pending → ContainerCreating → Running, but its readiness probe will initially report “not ready”.
While not Ready the pod does not receive traffic — it may be pulling model weights and initializing.
Once the readiness probe passes, the pod flips to Ready and joins the inference pool.
Only after the replacement is Ready does an old pod begin Terminating, drain in‑flight work, and exit.

This one‑by‑one turnover preserves capacity throughout the update.

3) Prometheus: verifying ready endpoints

From Prometheus we can inspect what the router sees for ready endpoints. The metric below shows how many ready endpoints the router sees for the quickstart-epp service:

llm_d_epp_ready_endpoints
llm_d_epp_ready_endpoints{container="epp", endpoint="http-metrics", instance="10.244.223.194:9090", job="quickstart-epp", name="quickstart", namespace="llm-d", pod="quickstart-epp-5f6cbf8c4-tcl7q", service="quickstart-epp"} 2

That count may wobble during the handover but — crucially — it never reaches 0. The rollout strategy ensures the system never reduces available capacity before a replacement is ready.

Observing the user stream during turnover

Back on the user terminal you should continue to see requests succeed:

request 101: 200
request 102: 200
request 103: 200
request 104: 200
...

No gaps, no error fingerprints — the update completed transparently from the callers’ perspective.

Why this is safe: three cooperating mechanisms

Mechanism	Role in zero-downtime updates	Notes / links
Rolling update strategy	Replaces pods one at a time so total capacity does not drop below the configured target	See Kubernetes rolling update docs: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment
Readiness gating	New pods are invisible to the router until the model is fully initialized and can answer requests	Readiness probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Router tracking	Router tracks pool membership live and only sends traffic to Ready endpoints	Router consults readiness-aware metrics (e.g., from Prometheus)

Together these ensure an update happens with no observable interruption: pods ready only when healthy, router only sends traffic to ready endpoints, and the rolling strategy prevents capacity dips. That’s day‑two operations in miniature — queues hold the line, autoscaling and readiness signals are trusted, and change leaves no fingerprints.

Quick command reference

Purpose	Command
Stream test requests	See the user loop above
Find decode deployment	`kubectl get deploy -n llm-d -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}'`
Trigger rollout (env var)	`kubectl set env deploy/<name> -n llm-d DEMO_ROLLOUT=v42`
Watch pods	`kubectl get pods -n llm-d -w`

Links and references

Kubernetes rolling updates: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment
Readiness probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Prometheus: https://prometheus.io/

A quick tip: run the user heartbeat and the operator rollout in separate terminals to see the real‑time effect. Use Prometheus or your router’s metrics to validate that ready endpoint counts never drop to zero.

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond

Demo A Rolling Update Under Live Traffic

1) Users: a steady heartbeat of requests

2) Operator: rolling the deployment

3) Prometheus: verifying ready endpoints

Observing the user stream during turnover

Why this is safe: three cooperating mechanisms

Quick command reference

Links and references

Watch Video

​1) Users: a steady heartbeat of requests

​2) Operator: rolling the deployment

​3) Prometheus: verifying ready endpoints

​Observing the user stream during turnover

​Why this is safe: three cooperating mechanisms

​Quick command reference

​Links and references

Watch Video

1) Users: a steady heartbeat of requests

2) Operator: rolling the deployment

3) Prometheus: verifying ready endpoints

Observing the user stream during turnover

Why this is safe: three cooperating mechanisms

Quick command reference

Links and references