Explains upgrading an approximate router prefix-cache scorer to a precise live reported-index that tracks pod cache events to improve routing accuracy and reduce wasted cache misses.
When we first read the router configuration, one small detail stood out and I promised we’d return to it. That detail is the difference between an approximate prefix-cache scorer and a precise, reported-index scorer — and how that difference matters under real cache pressure.The prefix-cache scorer currently running on our cluster is the approximate flavor. It does not query pods for their actual contents. Instead, it keeps a ledger of what it routed where and predicts cache residency from that history. For many workloads that heuristic is good enough.
This article explains where that heuristic breaks down and what the precise upgrade looks like.What the approximate scorer actually knows is simple: it keeps a little router ledger that says, “I sent this prefix to that pod, so that pod probably still has the cached work.” Every word of that sentence is doing honest work — except one.
Probably.The router ledger and the pod’s actual memory can drift apart. Typical causes:
Eviction. When memory fills, pods evict least-recently-used blocks without reporting to the router; the router still believes the block exists.
Asymmetric bookkeeping. The router chops traffic into blocks by its own rules, which might differ from the pod’s cache layout.
Router restarts. When a router restarts, its ledger starts blank and must relearn associations from observed traffic.
The result is the “routing perfectly to nothing” failure: the router directs requests to a pod expecting a cached block, arrives at an empty room, and the system pays the full prefill cost anyway.The prefix-aware router closes that gap in the most direct way possible: stop guessing; start listening.Model servers already publish KV events when the cache changes — block created, moved, or evicted. Small, fast notifications stream out as the cache mutates. The llm-d: A Deep Dive system collects that stream into a live, reported index so the router’s picture becomes a copy of the cache’s reported state instead of a prediction.Which blocks exist on which pod — and, once tiering is enabled, on which memory tier — is then known to the router:
live block indexblk-7f2 pod-0 GPUblk-3a9 pod-2 RAMblk-c14 pod-2 disk
When a request arrives, the scorer checks the prompt against blocks recorded as present in the live index, including recently reported evictions. Because the index reflects reported state, ghost entries are greatly reduced.
What does that buy you in practice?
Higher effective hit-rate. The approximate scorer misses reuse it never observed and sometimes routes to blocks that have already been evicted. The precise scorer sees shared prefixes wherever they actually exist and avoids routing to empty rooms.
More useful score weight. The same “three-slot” score and weight can deliver more real cache hits rather than wasted routes.
Below is a concise comparison.
Scoring flavor
Typical failure modes
Practical outcome
Approximate
Misses cross-client reuse; trusts stale ledger entries; sensitive to router restarts
Lower effective hit rate; wasted prefill work
Precise (live index)
Dependent on event reporting; knowledge is as fresh as reports arrive
Higher true hit rate; avoids routing to evicted blocks
The honest status: the optimized baseline and our cluster currently run the approximate flavor because that’s what ships by default today. The project has indicated the precise flavor is where the default is heading.
Turning the precise scorer on is a configuration decision: point the same plugin file at the etcd-fed live index instead of the built-in guess. Nothing about the fleet must change, and demos run the same — for traffic patterns where the approximate scorer already works well, you’ll see little visual difference.
Enabling precise scoring is a configuration switch: update the router plugin to read the etcd-fed live index (the KV event stream) instead of the local heuristic ledger. This keeps the deployment surface unchanged while improving routing accuracy.
But even perfect, up-to-date knowledge cannot undo an eviction that already happened. The precise index will tell the router immediately when a block is evicted — which prevents wasted routing going forward — but it won’t resurrect the discarded block.If you want cached work to survive memory pressure, the blocks themselves need durable placement: tiering. The foundation sketches this idea as memory tiers (GPU → RAM → disk), so when the hottest tier fills, blocks can demote to lower tiers instead of being discarded outright. Combined with the live index, tiering makes routing both accurate and resilient.
Precise reporting reduces ghost routes, but it does not prevent evictions. To reduce lost work under pressure, enable block tiering so cached data can demote instead of being immediately discarded.
