Explains how the Endpoint Picker scores and selects pods using cache affinity and load-awareness to reduce context reloads and improve LLM serving throughput.
So far we’ve covered how to stand the system up and send it a prompt: the request reaches the gateway, the scheduler picks a pod, and an answer returns. The middle step — the scheduler — does most of the hard work, so let’s expand that.The component we’ve been calling the scheduler (or the “small brain” out front) has a proper name: the Endpoint Picker (EPP). Its job is exactly that — pick an endpoint (pod) from a fleet that can serve the request.If you’ve run services in production you’re familiar with load balancers: they watch server health and steer new requests toward the least-busy server. It would be natural to assume an LLM-serving system behaves the same way (just with fancier metrics). The key difference — and the source of much of the throughput improvement here — is that it does not. Instead, the EPP uses a per-request, claim-based scoring model that factors in cache affinity and load to minimize expensive context reloads.Typical topology:
A fleet of pods (prefill pods, decode pods, cache-aware instances).
A Kubernetes gateway that takes requests off the network.
The EPP running beside the gateway to decide which pod gets each request.
Inside the EPP, a request flows through three high-level stages:
Stage
Purpose
Outcome
Request handling
Parse headers and JSON body into a normalized request object
A standardized request object for downstream stages
Decide if the request should run now, be throttled, or wait
Request scheduling
Choose which pod will serve the request
One pod selected and the request forwarded to it
The flow-control stage contains several important policies (rate limiting, prioritization, concurrency limits). These policies are complex and are beyond the scope of this article.
Request scheduling itself breaks down into three sub-steps. Conceptually, for a chat request like:
POST /v1/chatModel: llamaBody: {"messages": [...]}
the EPP does the following:
Filter: remove pods that cannot handle the request (wrong role, not ready, model not loaded, insufficient GPU memory, etc.). From an initial pool of 40 candidates you might end up with ~12 suitable pods.
Score: run a set of scoring plugins on each remaining candidate and compute numeric scores.
Select: pick the highest-scoring pod and forward the request.
Scoring is plugin-driven. The EPP uses a shortlist of scorer plugins that each assert a numeric claim about a pod at that moment. Typical plugins include:
Plugin
What it measures
Effect
Route-by-saved-work (KV cache)
Whether the pod already holds parts of the conversation or other cached context
Gives higher score to pods that already have relevant context (cache affinity)
Load-based scoring
Current CPU/GPU load, queue length, or in-flight request count
Penalizes pods under heavy load so less-busy pods score higher
Each plugin runs over the same filtered list and produces a numeric contribution. The EPP adds these contributions to produce a final score per pod, then selects the highest-scoring pod.
Why is this different from a plain load balancer? Consider a simple example: pod 0 is busier than pod 1. A classic load balancer would pick pod 1 to keep load balanced. But if pod 0 already contains most of your conversation in its KV cache, sending the request to pod 0 avoids an expensive context reload. The EPP will add the cache-affinity score to the load score, and pod 0 can end up with a higher total score — so the router intentionally routes to the busier pod to skip the reload delay.This behavior trades short-term load balance for lower overall latency for the request being routed — it’s an intelligent trade-off that produces the measured throughput gains.
Conceptually: the EPP is a scoring loop over a filtered list. Each numeric score is a claim about a pod in that instant — how busy it is, what context it holds, queue depth, etc. These real-time claims are summed into a single value that represents how well a pod can serve the request right now.
This claim-based scoring is the source of the numeric speedup: the router can route your request to the pod that already “remembers” you, avoiding reloading large contexts from backing storage. That optimization is not merely a better load balancer — it’s an intelligent router that combines cache affinity with load-awareness to make per-request decisions.
Where do the numeric score components come from? Each score is derived from metrics that describe the fleet at that moment: per-pod load, queue lengths, cache contents, age of cached items, and other telemetry. The next section (not covered here) explains how the EPP gathers and maintains these metrics so the scorer can make accurate claims.
