Tiered caching and live block indexing that move LLM conversation state between GPU RAM and disk to avoid recomputation and enable reliable routing.
The last session ended on an uneasy note.All of that clever routing depends on one assumption: the saved work — the KV cache — is still available when you return. On a single machine, that saved work lives in one place:
That one place is the GPU’s memory — the same place where space is most desperately short.What happens when that memory fills up? To answer, picture the GPU’s internal layout: the model’s weights take a fixed share of memory, and everything left over stores the scratchpads — the per-conversation cached “saved work.” That layout is repeated for every conversation on the GPU. As load grows, the leftover space shrinks. When a new request needs room, the server must pick something to discard. The practical choice is eviction of the item that has gone the longest without being used.
This eviction policy is called Least Recently Used (LRU).
Whoever has been quiet the longest loses their seat. Often that “who” is your own conversation — the one from a few minutes ago you were about to continue. The GPU silently discards that conversation’s scratchpad blocks; there’s no error or notice. The next time you send a message, the server must recompute the prefill across the entire conversation and rebuild those blocks — precisely the costly work the cache was intended to avoid.This behaviour is not a bug. With one limited, high-speed tier for saved work, a server has only two options: evict something or refuse new requests. Refusing requests is not generally acceptable, so eviction is the unavoidable practical choice. What eviction discards are just blocks of numbers (tensors) sitting in the wrong place — not broken work or incomplete answers.Computers already provide multiple kinds of memory. The GPU is the penthouse: blistering fast but very small. One floor down is the machine’s RAM — slower but far roomier. Beneath that is disk — slower still, but practically limitless.Because the saved work is just tensors, moving a block down one floor (GPU → RAM → disk) and later pulling it back up costs significantly less than recomputing it (the prefill pass). Creating the block is the expensive part; moving bytes between tiers is cheap by comparison.To exploit that difference, llm-d: A Deep Dive implements a tiered cache. When the GPU fills, blocks are demoted rather than destroyed:
If the GPU is full, blocks are moved down to RAM.
If RAM fills, blocks are moved down to disk.
When a conversation becomes active again, its blocks get promoted back up to the tier they need.
Each tier holds the data it’s best suited for: the GPU keeps the white-hot working set, RAM holds recently active conversations, and disk stores the long tail that might still return.
A quick reference:
Tier
Typical role
Performance tradeoff
GPU
White-hot working set (active responses)
Fastest execution, smallest capacity
RAM
Recently active conversations
Slower than GPU but much larger; good for short-term retention
Disk
Long tail / seldom-used conversations
Slowest, essentially unlimited; still much faster than recomputing prefill for long contexts
Here’s the payoff: even loading blocks from the slowest tier (disk) and copying them back into GPU memory is typically much faster than redoing the entire prefill computation. The longer the conversation, the larger the savings. Measurements from the llm-d: A Deep Dive project show long prompts restored from storage can be up to nearly 17× faster than recomputing the prefill.
Because blocks move between tiers and occasionally expire entirely, the location of any conversation’s saved work is constantly changing. A router that relied on stale information could send you to a pod that just finished forgetting you. To prevent that, the model servers publish every state change.Every time a block is created, moved, or evicted, the pod publishes an event. An llm-d: A Deep Dive index maintains a live map of which blocks exist, which pod holds them, and on which tier (GPU, RAM, or disk) they currently reside.
The router computes routing scores based on what the live index reports as actually present, not on historical assumptions. That keeps routing accurate as blocks migrate.
Putting the pieces together:
A model is a large file of numbers that needs a GPU to execute.
Generation proceeds token-by-token in two phases: the prefill (conditioning on history) and the decode (autoregressive steps).
The scratchpad makes streaming feel instant. Batching keeps it affordable. Sharding makes very large models feasible.
In a fleet, routing becomes complex. llm-d: A Deep Dive solves this with a router that knows where saved work currently lives and a tiered cache that survives temporary space shortages.
Every piece of this design is a response to a constraint created by the piece before it. Next up is demonstration: a real GPU cluster, real pods, and live requests so you can observe how the system behaves in practice.
Tiered caching (GPU → RAM → disk) plus a live block index delivers both strong performance and predictable routing: copying blocks between tiers is cheap compared with recomputing them, and the router always knows where to send requests.
