Demo Watch One Request Happen

The model server exposes three important per-request guarantees: a path (request routing), a stream (token-by-token decode), and a KV scratchpad (an in-memory cache that remembers intermediate state). In this walkthrough we observe all three on a single model pod so you can see how prefill, decode (streaming), and the KV scratchpad interact in real time. We deliberately bypass the router and communicate directly with a single pod. When multiple pods sit behind a router you cannot control which pod answers, which prevents us from watching a single pod’s internal metrics. Bypassing the router ensures the request hits the pod we target. What you will learn

How to select and port-forward to a single model pod.
How to watch runtime gauges that reveal request lifecycle and KV cache usage.
How to issue a streaming completion and observe prefill, streaming decode, and scratchpad behavior.

Prerequisites

kubectl configured to the cluster where the model pods run.
A model pod labeled llm-d.ai/role=decode is running in the namespace you choose.
curl and watch installed on your local machine.

Prepare your shell

Open a fresh terminal and set a namespace variable. Then pick one pod labeled as a decode pod (we pick the first one returned).

# set namespace (replace with your namespace if different)
export NAMESPACE=llm

# pick the first pod labeled as a decode pod
export POD=$(kubectl get pods -n $NAMESPACE -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}')

Make sure you export the correct NAMESPACE and that the pod you pick is a decode pod. Choosing the wrong namespace or pod may cause commands to fail or target the wrong service.

Wire a straight line to the pod (port-forward)

Port-forward the model server’s HTTP port (8001) to localhost so you can query the pod directly. Keep this terminal open while you run the other commands.

kubectl port-forward -n $NAMESPACE $POD 8001:8001

Watch the pod’s runtime gauges

The pod exposes Prometheus-style metrics on the same port. In a second terminal run a short watch that queries metrics every second and displays two values we care about: vllm:num_requests_running and vllm:kv_cache_usage_perc.

watch -n 1 "curl -s http://localhost:8001/metrics | egrep 'vllm:num_requests_running|vllm:kv_cache_usage_perc'"

When the pod is idle you should see something like:

Every 1.0s: curl -s http://localhost:8001/metrics | egrep 'vllm:num_requests_running|vllm:kv_cache_usage_perc'  Thu Oct 20 12:00:00 2026

vllm:num_requests_running{engine="0",model_name="Qwen/Qwen2.5-0.5B-Instruct"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen2.5-0.5B-Instruct"} 0.0

At rest, both gauges should read zero. Quick reference — what these metrics mean

Metric	What it indicates	Example value
`vllm:num_requests_running`	Number of active requests currently being processed by the pod.	`0.0` (idle) or `1.0` (one active request)
`vllm:kv_cache_usage_perc`	Percentage of the model’s per-request KV scratchpad currently used. This tracks memory reserved for cached key/value pairs while decoding.	`0.0` (empty) up to `100.0`

Issue a streaming completion to the pod

Back in the terminal where you started the port-forward, send a streaming completion to the pod. Note stream is true. We intentionally give a large max_tokens budget to keep the decode loop running long enough to observe prefill and decode on CPU.

curl -sN http://localhost:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "prompt": "Explain, in detail, why serving a large language model is hard. Cover the memory it takes, the compute it takes, and why serving many users at once makes everything harder.",
    "max_tokens": 1000,
    "stream": true
  }'

What you will observe

Short silence after sending the request: this is the prefill phase where the model ingests and processes the full prompt in parallel before emitting tokens.
After the pause, tokens stream back as SSE-style chunks. While tokens stream, switch to the metrics watch window and observe:
- vllm:num_requests_running rises from 0 to 1.
- vllm:kv_cache_usage_perc increases as the KV scratchpad fills during decoding.
When the request completes the metrics return to zero, demonstrating the request lifecycle.

Example of streaming SSE chunks (truncated and simplified for clarity):

data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" on","finish_reason":null}]}
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" the","finish_reason":null}]}
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" number","finish_reason":null}]}
...
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":"","finish_reason":"stop"}]}
data: [DONE]

Lifecycle recap

Prefill: the initial pause while the model ingests the prompt.
Decode: token-by-token streaming output (SSE chunks).
Scratchpad: the in-memory KV cache used during decoding; visible via vllm:kv_cache_usage_perc.

On GPU hardware in production, the same prefill + decode loop executes much faster (hundreds of tokens per second). Running on CPU slows the loop so you can observe each stage; the behavior and guarantees remain the same. The KV cache metric reflects real memory usage — not an abstract concept.

What we did and what’s next We bypassed the router to ensure the request reached a single pod so we could observe its internal metrics. With a fleet behind a router you cannot choose the responding pod — the router decides. The router may or may not use per-pod cache information when making routing decisions; that behavior depends on router configuration. Next steps

Reintroduce the router and send the same prompt repeatedly to observe request distribution across the fleet.
Inspect the router configuration (routing “dials”) to understand how it considers cached state when deciding where to route requests.

Links and references

Kubernetes Basics
kubectl port-forward documentation
SSE (Server-Sent Events) overview: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond

Demo Watch One Request Happen

Watch Video