> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo The Same Prompt Lands on the Same Server

> Demonstrates how a cluster router and per-pod prefix cache route repeated prompts to the same model server, reusing cached token blocks to reduce latency and CPU

This demo shows how the cluster's router and per-pod prefix cache work together to route repeated prompts back to the same server and reuse previously-computed prefix blocks. The router’s endpoint picker prefers pods that already hold saved work, and the model server’s prefix cache stores fixed-size token blocks so repeated prompts can skip expensive prefill work.

Why this matters: for frequent, identical or highly similar requests, routing to saved work reduces latency and CPU usage by reusing cached prefix blocks instead of redoing the full prefill for every request.

## Key detail: cache block size

The prefix cache stores data in fixed-size blocks. On this cluster the model server uses 128-token blocks. Any prompt shorter than one block will never fill the cache and therefore will not register as a prefix hit, even if you send it many times. Make sure repeated prompts are long enough to cross that block boundary when testing prefix reuse.

Short prompts (less than the configured block size — 128 tokens in this demo) will not produce prefix cache hits. Cold requests seed the cache; only repeated requests that match the cached prefix produce hits.

## Prepare a realistic repeated prompt

Here’s a realistic, reusable lead-in used in many runbooks. We store it in an environment variable so we can send it repeatedly:

```bash theme={null}
STATEMENT='Our platform team is preparing a runbook for operating large language model services on Kubernetes. The runbook should explain how requests flow from the gateway to the model servers, how the scheduler decides which server should handle each request, and how the key value cache keeps repeated conversations fast. It should also describe what happens when memory fills up, how rolling updates avoid dropping traffic, and which dashboards the on-call engineer should check first when latency rises. Please draft the opening section of this runbook. Keep the tone practical, assume the reader knows Kubernetes but is new to model serving, use short paragraphs, and include one note about why batching keeps serving costs down.'
```

## Inspect per-pod prefix cache hits in Prometheus

Before you send requests, confirm there are no prefix cache hits yet by querying Prometheus. The prefix cache hit metric is exposed per pod, so you can see which pod (if any) has cached the statement.

* Prometheus docs: [https://prometheus.io](https://prometheus.io)

A useful query to run:

```text theme={null}
sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))
```

If the result is all zeros, no pod has cached the statement yet — good starting point.

## Seed the cache and repeat the request

Send the same request repeatedly through the router. The first request will pick a pod (router decides) and that pod will perform the full prefill and populate its cache. Subsequent identical requests should be routed to the pod holding those cached blocks and will skip heavy prefill work.

Run this loop locally (example uses curl + jq):

```bash theme={null}
for i in $(seq 1 6); do
  ANSWER=$(curl -s http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"prompt\":\"${STATEMENT}\",\"max_tokens\":30}" \
    | jq -r '.choices[0].text' | tr '\n' ' ')
  echo "answer ${i}: ${ANSWER}"
done
```

Sample returned answers (truncated for readability):

```text theme={null}
answer 1:  **Welcome to the Kubernetes Model Serving Runbook**  As you navigate through our Kubernetes environment, your journey will be guided by the principles of efficient service orchestration...
answer 2:  Here's an example: ---  # Running Large Language Model Services on Kubernetes  ## Overview This runbook provides guidance on deploying and managing a large language...
```

## Verify cache reuse in Prometheus

After sending the repeated requests, re-run the Prometheus query. The expected behavior is that one pod’s prefix cache hit counter increases while the others remain at zero — confirming both routing-to-saved-work and cache reuse.

Example query result:

```text theme={null}
sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))

{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-g7kkb"} 1347.3684210526314
```

What this proves:

* The router consistently routed repeats to the same pod (routing to saved work).
* The receiving pod reused cached prefix blocks on every repeated request (prefix cache hits).

The very first of our six sends will never show up as a prefix hit because there was nothing cached yet — cold requests seed the cache. Only the repeated requests generate the subsequent hits.

## Quick reference

| Item | Why it matters | Example / Query |
| - | - | - |
| Prometheus query to sum hits per pod | Shows which pod served repeated cached prefixes | `sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))` |
| Prometheus metric name | Metric exposed by vLLM model servers | `vllm:prefix_cache_hits_total` |
| Block size caution | Prompts shorter than the block size never fill the cache | Block size = 128 tokens (demo) |

## Notes on the opposite scenario

If you send many different prompts (e.g., 30 different prompts across the fleet), you should not expect cache reuse — that’s a different test case. High cardinality of unique prompts means less cache benefit and different scaling/latency characteristics, which should be tested separately.

## References

* Prometheus — [https://prometheus.io](https://prometheus.io)
* For routing and scheduler behavior, consult your cluster’s gateway/router docs and the model-serving scheduler documentation used in your environment.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/a6ab4920-a1d0-4f6a-aa81-fa7b8f8ac0e9/lesson/a4ab256b-6103-48c2-b97c-9088d48d237c)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.