Skip to main content
Last time we finished with a routing plan. Before we run the experiment, read the rules. The router can observe every pod’s queue length, memory usage, and cache signals. Those observations are not used ad-hoc — the router follows a small, human-editable configuration delivered via a Kubernetes ConfigMap. In this lesson we open that file and read the router’s instructions.
No installation or feature flag is required. The quickstart-epp deployment you already ran includes an active, cache-aware, and load-aware routing pipeline — the intelligence has been on since the first request.
A stylized network diagram titled "The Router's Rules" showing a Router component distributing traffic to three pods (pod 0, pod 1, pod 2) with per-pod cards showing queue counts and utilization bars. A legend at left indicates the router is cache-aware and load-aware.
Inspect the ConfigMaps in the llm-d namespace and find the quickstart-epp ConfigMap that contains the plugin configuration:
Open the quickstart-epp ConfigMap to view the router plugin pipeline. The file expresses the three-stage pipeline (filter → score → select) and lists which scorers are active plus the scheduling profile that assigns numeric weights to each scorer:
Read the plugin names carefully — these are the actual scorers the router runs. The profile weights encode the router’s behavior and priorities: higher weight means the associated signal influences routing decisions more strongly. Below is a concise summary of the configured scorers and weights for quick reference. The star of this configuration is the prefix-cache-scorer (weight 3). It rewards pods where saved work for the incoming prompt is most likely to exist — reducing latency and compute by allowing the router to skip or shorten prefill work. In this quickstart the cluster runs the approximate flavor: instead of querying every pod for exact cache contents it uses routing history and heuristics to predict where prompts likely live. The precise flavor, which consumes live CacheEvent updates from pods, is a higher-fidelity mode we’ll cover later.
After cache-aware scoring, the router applies three load-oriented scorers. Each of these has weight 2 in the shown scheduling profile:
  • queue-scorer: rates pods by current queue length (shorter is better).
  • kv-cache-utilization-scorer: rates pods by how much of their scratchpad memory is used (more free memory is better).
  • no-hit-lru-scorer: biases routing for “cold” or previously-unseen prompts to spread them across pods.
Roomier memory and shorter queues are both desirable.
A dark-themed infographic titled "Three Load Scorers" showing three scorer modules (e.g., queue-scorer, kv-cache-utilization-scorer, no-hit-lru-scorer) with icons, multiplier badges and horizontal rating bars on the left. On the right are three pod boxes labeled pod 0, pod 1, and pod 2 with small user icons and queue counts.
The clever load-oriented scorer is no-hit-lru-scorer. It only influences routing for “strangers” — prompts that have no saved work anywhere in the cluster. Since every newcomer will seed a fresh cache trail wherever it lands, the router intentionally spreads newcomers so new cache entries are distributed instead of all landing on one pod. The no-hit-lru-scorer steers a cold request toward the pod that received a cold request least recently.
A dark-themed diagram titled "Spread the Newcomers." On the left is a node labeled "no-hit-lru-scorer ×2" connected by pink lines to three pod cards on the right (pod 0, pod 1, pod 2), each showing progress bars and percentage readouts.
How the final decision is made: for every surviving pod the router computes a numeric rating from each scorer, multiplies that rating by the scorer’s configured weight, and sums the weighted values into a single score. The pod with the highest score wins the request. In our profile cache reuse (weight 3) has higher influence than any single busyness signal (weight 2), which means avoiding prefill work is often worth accepting a slightly higher queue.
A dark-themed dashboard titled "Put the Numbers Together" showing rows of icons and numeric values with highlighted totals (e.g. 3.8, 3.6, 2.0). On the right are three pod cards (pod 0, pod 1, pod 2) each with a progress bar and percentage.
From this single YAML profile we can state two clear, testable promises that the cluster should uphold:
  • Promise 1: Repeated prompts the cluster has seen before should be routed back to the same pod repeatedly (cache benefit outweighs a single busyness metric).
  • Promise 2: A stream of brand-new prompts (no saved work anywhere) should be spread across pods, due to load scorers and the newcomer spreader.
Two predictions driven by four lines of YAML.
A dark-themed infographic titled "Two Testable Promises" comparing "Promise 1" (Same pod) and "Promise 2" (Spread). To the right are three pod cards (pod 0, pod 1, pod 2) with colored load bars, small icons, and percentage/queue indicators.
Time to test the first promise.
If your cluster uses the approximate flavor, scoring is based on history and heuristics rather than live cache indices. For the precise, CacheEvent-backed flavor, the router consults real-time cache membership published by each pod.
Further reading and references

Watch Video