Skip to main content
This demo shows how the cluster’s router and per-pod prefix cache work together to route repeated prompts back to the same server and reuse previously-computed prefix blocks. The router’s endpoint picker prefers pods that already hold saved work, and the model server’s prefix cache stores fixed-size token blocks so repeated prompts can skip expensive prefill work. Why this matters: for frequent, identical or highly similar requests, routing to saved work reduces latency and CPU usage by reusing cached prefix blocks instead of redoing the full prefill for every request.

​
Key detail: cache block size

The prefix cache stores data in fixed-size blocks. On this cluster the model server uses 128-token blocks. Any prompt shorter than one block will never fill the cache and therefore will not register as a prefix hit, even if you send it many times. Make sure repeated prompts are long enough to cross that block boundary when testing prefix reuse.
Short prompts (less than the configured block size — 128 tokens in this demo) will not produce prefix cache hits. Cold requests seed the cache; only repeated requests that match the cached prefix produce hits.

​
Prepare a realistic repeated prompt

Here’s a realistic, reusable lead-in used in many runbooks. We store it in an environment variable so we can send it repeatedly:

​
Inspect per-pod prefix cache hits in Prometheus

Before you send requests, confirm there are no prefix cache hits yet by querying Prometheus. The prefix cache hit metric is exposed per pod, so you can see which pod (if any) has cached the statement. A useful query to run:
If the result is all zeros, no pod has cached the statement yet — good starting point.

​
Seed the cache and repeat the request

Send the same request repeatedly through the router. The first request will pick a pod (router decides) and that pod will perform the full prefill and populate its cache. Subsequent identical requests should be routed to the pod holding those cached blocks and will skip heavy prefill work. Run this loop locally (example uses curl + jq):
Sample returned answers (truncated for readability):

​
Verify cache reuse in Prometheus

After sending the repeated requests, re-run the Prometheus query. The expected behavior is that one pod’s prefix cache hit counter increases while the others remain at zero — confirming both routing-to-saved-work and cache reuse. Example query result:
What this proves:
  • The router consistently routed repeats to the same pod (routing to saved work).
  • The receiving pod reused cached prefix blocks on every repeated request (prefix cache hits).
The very first of our six sends will never show up as a prefix hit because there was nothing cached yet — cold requests seed the cache. Only the repeated requests generate the subsequent hits.

​
Quick reference

​
Notes on the opposite scenario

If you send many different prompts (e.g., 30 different prompts across the fleet), you should not expect cache reuse — that’s a different test case. High cardinality of unique prompts means less cache benefit and different scaling/latency characteristics, which should be tested separately.

​
References

  • Prometheus — https://prometheus.io
  • For routing and scheduler behavior, consult your cluster’s gateway/router docs and the model-serving scheduler documentation used in your environment.

Watch Video