> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Watch One Request Happen

> Walkthrough showing how to port‑forward to a single model pod, stream a completion, and observe prefill, token decoding, and KV scratchpad metrics in real time.

The model server exposes three important per-request guarantees: a path (request routing), a stream (token-by-token decode), and a KV scratchpad (an in-memory cache that remembers intermediate state). In this walkthrough we observe all three on a single model pod so you can see how prefill, decode (streaming), and the KV scratchpad interact in real time.

We deliberately bypass the router and communicate directly with a single pod. When multiple pods sit behind a router you cannot control which pod answers, which prevents us from watching a single pod's internal metrics. Bypassing the router ensures the request hits the pod we target.

What you will learn

* How to select and port-forward to a single model pod.
* How to watch runtime gauges that reveal request lifecycle and KV cache usage.
* How to issue a streaming completion and observe prefill, streaming decode, and scratchpad behavior.

Prerequisites

* kubectl configured to the cluster where the model pods run.
* A model pod labeled `llm-d.ai/role=decode` is running in the namespace you choose.
* curl and watch installed on your local machine.

1. Prepare your shell

Open a fresh terminal and set a namespace variable. Then pick one pod labeled as a decode pod (we pick the first one returned).

```bash theme={null}
# set namespace (replace with your namespace if different)
export NAMESPACE=llm

# pick the first pod labeled as a decode pod
export POD=$(kubectl get pods -n $NAMESPACE -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}')
```

Make sure you export the correct `NAMESPACE` and that the pod you pick is a decode pod. Choosing the wrong namespace or pod may cause commands to fail or target the wrong service.

2. Wire a straight line to the pod (port-forward)

Port-forward the model server's HTTP port (8001) to localhost so you can query the pod directly. Keep this terminal open while you run the other commands.

```bash theme={null}
kubectl port-forward -n $NAMESPACE $POD 8001:8001
```

3. Watch the pod's runtime gauges

The pod exposes Prometheus-style metrics on the same port. In a second terminal run a short watch that queries metrics every second and displays two values we care about: `vllm:num_requests_running` and `vllm:kv_cache_usage_perc`.

```bash theme={null}
watch -n 1 "curl -s http://localhost:8001/metrics | egrep 'vllm:num_requests_running|vllm:kv_cache_usage_perc'"
```

When the pod is idle you should see something like:

```bash theme={null}
Every 1.0s: curl -s http://localhost:8001/metrics | egrep 'vllm:num_requests_running|vllm:kv_cache_usage_perc'  Thu Oct 20 12:00:00 2026

vllm:num_requests_running{engine="0",model_name="Qwen/Qwen2.5-0.5B-Instruct"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen2.5-0.5B-Instruct"} 0.0
```

At rest, both gauges should read zero.

Quick reference — what these metrics mean

| Metric | What it indicates | Example value |
| - | - | - |
| `vllm:num_requests_running` | Number of active requests currently being processed by the pod. | `0.0` (idle) or `1.0` (one active request) |
| `vllm:kv_cache_usage_perc` | Percentage of the model's per-request KV scratchpad currently used. This tracks memory reserved for cached key/value pairs while decoding. | `0.0` (empty) up to `100.0` |

4. Issue a streaming completion to the pod

Back in the terminal where you started the port-forward, send a streaming completion to the pod. Note `stream` is `true`. We intentionally give a large `max_tokens` budget to keep the decode loop running long enough to observe prefill and decode on CPU.

```bash theme={null}
curl -sN http://localhost:8001/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "prompt": "Explain, in detail, why serving a large language model is hard. Cover the memory it takes, the compute it takes, and why serving many users at once makes everything harder.",
    "max_tokens": 1000,
    "stream": true
  }'
```

What you will observe

* Short silence after sending the request: this is the prefill phase where the model ingests and processes the full prompt in parallel before emitting tokens.
* After the pause, tokens stream back as SSE-style chunks. While tokens stream, switch to the metrics watch window and observe:
  * `vllm:num_requests_running` rises from 0 to 1.
  * `vllm:kv_cache_usage_perc` increases as the KV scratchpad fills during decoding.
* When the request completes the metrics return to zero, demonstrating the request lifecycle.

Example of streaming SSE chunks (truncated and simplified for clarity):

```text theme={null}
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" on","finish_reason":null}]}
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" the","finish_reason":null}]}
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":" number","finish_reason":null}]}
...
data: {"id":"cpl-a14210fc78afb494","object":"text_completion","created":1786727651,"model":"Qwen/Qwen2.5-0.5B-Instruct","choices":[{"index":0,"text":"","finish_reason":"stop"}]}
data: [DONE]
```

Lifecycle recap

* Prefill: the initial pause while the model ingests the prompt.
* Decode: token-by-token streaming output (SSE chunks).
* Scratchpad: the in-memory KV cache used during decoding; visible via `vllm:kv_cache_usage_perc`.

On GPU hardware in production, the same prefill + decode loop executes much faster (hundreds of tokens per second). Running on CPU slows the loop so you can observe each stage; the behavior and guarantees remain the same. The KV cache metric reflects real memory usage — not an abstract concept.

What we did and what’s next

We bypassed the router to ensure the request reached a single pod so we could observe its internal metrics. With a fleet behind a router you cannot choose the responding pod — the router decides. The router may or may not use per-pod cache information when making routing decisions; that behavior depends on router configuration.

Next steps

* Reintroduce the router and send the same prompt repeatedly to observe request distribution across the fleet.
* Inspect the router configuration (routing "dials") to understand how it considers cached state when deciding where to route requests.

Links and references

* [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
* [kubectl port-forward documentation](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
* SSE (Server-Sent Events) overview: [https://developer.mozilla.org/en-US/docs/Web/API/Server-sent\_events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/cb6d2710-3350-448e-bc67-d78835a558a9/lesson/93316b28-c63e-42c5-b152-543ef1c919e3)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.