Describes predictive routing that predicts per-pod TTFT and TPOT with XGBoost trained on live requests to minimize latency, noting KV cache eviction limits effectiveness.
Can a router reason about what will happen to a pod in the next few hundred milliseconds — rather than only observing the instant it was sampled? Yes. Predictive routing uses per-request learning to estimate latency before you send your request, and routes to minimize the delay you actually experience.Scenario: you have multiple decode pods. At sampling time Pod A looks best — its queue is empty and half its memory is free — so the EPP (edge pick-and-place) routes your request there. What the EPP can’t see is that 80 ms earlier another user started parsing a 40‑page document on Pod A. That work is still in prefill, hasn’t yet shown up in the scraped queue depth, and
it hasn’t claimed the pod’s memory yet — but it will. So you were routed to the pod that looked emptiest but is about to be the busiest.This isn’t a metric bug: the snapshot was accurate for the sample moment. But sampling is reactive by nature — always one step behind events that are already in flight.
Instead of chasing heuristic signals like queue depth or memory pressure, ask the question that actually matters for user experience: how slow will this pod be for my request? Distill latency into two actionable numbers:
Metric
What it measures
Why it matters
TTFT
Time To First Token — the latency before the model emits its first token
Determines how long a user waits for any response at all
TPOT
Time Per Output Token — average inter‑token gap once streaming begins
Governs throughput and how snappy the stream feels (same as ITL — inter‑token latency)
Queue depth, memory pressure, and batch size are all proxies for these two measures.
llm-d (the routing system) therefore routes on predicted TTFT and predicted TPOT for each candidate pod, not on the stand‑ins.That raises the key question: how can you know those two numbers in advance?
You can’t compute TTFT and TPOT exactly from first principles — they depend on GPU model, how much of your prompt is already cached (KV cache), inflight requests, batching behavior, token counts, and many small hardware and workload interactions. But every completed request is an example: inputs in, TTFT/TPOT out. So llm-d learns to predict those numbers from live examples.How the predictive loop is built
The EPP pod gains two tiny sidecars:
a latency training server that converts every completed request into a training sample (prompt features, pod state at request time, measured TTFT and TPOT), and
a latency prediction server that reads the trained model and answers: “Given this prompt and this pod state, what are TTFT and TPOT?”
The training sample encodes:
pod telemetry (GPU type, approximate cache occupancy, inflight and queued work, recent batch sizes),
request features (token counts, prompt attributes),
and the measured TTFT/TPOT that resulted.
Those samples train an XGBoost model — a stack of decision trees that together form a fast, interpretable regressor.XGBoost (eXtreme Gradient Boosting) is a distributed, open‑source library for gradient‑boosted decision trees. For structured telemetry features like these, boosted trees often train faster and are easier to interpret than deep neural nets, while delivering accurate predictions.
The training server writes the latest model to a shared volume. The prediction sidecar loads that model and serves only one endpoint: given a candidate pod + prompt, return predicted TTFT and TPOT.
Operationally, the prediction server is a tiny sidecar inside each EPP pod. It serves per‑pod predictions from the XGBoost model that the training sidecar continuously retrains on live traffic. With per‑pod TTFT/TPOT estimates, the scheduler can compute an expected end‑to‑end latency and pick the pod that minimizes it — without hand‑tuning weights for queue depth, memory, and batch size.
On each incoming request, before selecting a target, the EPP asks the prediction server for each candidate pod and chooses based on the predicted TTFT and TPOT.
The loop then closes: the request runs, its real TTFT/TPOT are measured, and that measurement becomes a fresh training sample. The model continuously adapts to the live workload and hardware, removing the need for brittle manual tuning.
Measured impactGoogle tested this predictive routing against heuristic scorers (queue depth and memory scoring) and reported roughly 40% lower TTFT and ITL (inter‑token latency).
ITL is the same gap TPOT measures; the key difference is that the router predicted future latency instead of reading present metrics.
Prediction also enables policy‑driven routing: once the router can say Pod A ≈ 400 ms and Pod B ≈ 900 ms, you can attach latency targets and route toward SLOs rather than implicit heuristics. Prediction decides where a request should go.But this entire approach depends on one fragile resource: the KV cache — the saved state (your conversation, attention key/values) that lives in GPU memory. GPU memory is finite; when it fills, eviction happens.
Your conversation from four minutes ago can be evicted to make room for another user’s state.
If you route perfectly to a pod whose KV cache has already been evicted, you’ve routed perfectly to nothing.
So what happens when the scratchpad overflows?
Prediction improves routing by estimating end-to-end latency (TTFT and TPOT) from live, per-request signals, but it depends on cached state (the KV cache) being present on the chosen pod. Cache eviction remains a hard constraint: routing can only be as effective as the cache retention policy and available GPU memory allow.
