Skip to main content
The project’s strategy for handling cache overflow is best understood visually: think of memory as floors in a building. When the fastest floor (GPU or local memory) fills, items are demoted to a lower floor instead of being discarded; when activity returns, those items are promoted back up. This tiered approach reduces recomputation and can yield large speedups — in some tests the system measured nearly 17× faster responses on long prompts by avoiding repeated prefill work.
An infographic titled "Floors of Memory" showing a stacked hierarchy: GPU at the top labeled "fastest / smallest," RAM in the middle labeled "slower / roomier," and Disk at the bottom labeled "slowest / bottomless."
Tiering is not an extra component you install beside llm-d; it is a capability built into the model server. You enable and size it through the model server’s caching configuration. The server coordinates movement between floors using an offloading connector — a lightweight piece that sits next to the cache and routes block traffic between tiers. The connector is responsible only for moving blocks; it does not change their structure. Each block remains a fixed-size unit and can live on any tier. In production fleets the storage tier is typically a shared disk exposed to each node (for example, a Kubernetes PVC). The implementation is intentionally pluggable: the project ships a filesystem backend for the storage tier, and the connector accepts alternative caching engines from the ecosystem.
A dark-themed diagram titled "How the Blocks Move" showing an "Offloading connector" icon with arrows branching down to two labeled boxes: "File system (Storage tier)" on the left and "Other engines (Ecosystem)" on the right. The image appears to be a flowchart illustrating where blocks are offloaded.
Configuration basics — what you set for each tier:
  • Existence: enable or disable the tier.
  • Size: how much capacity to allocate (e.g., amount of RAM, disk path/size, or GPU memory reservation).
The storage tier can use a local filesystem backend or be replaced with alternative engines. Because the connector accepts multiple backends, you can prototype with local disk and later migrate to a networked cache or object store without changing how blocks are represented.
Demote, don’t delete. Once tiering is enabled the hierarchy organizes itself as expected:
  • The fast tier (GPU or local high-speed memory) contains the hot working set — the messages being answered right now.
  • The RAM tier holds recently active state that may be needed again in minutes.
  • The storage tier (shared disk) keeps the long tail: conversation histories, system prompts, and other state that’s unlikely to be hot but is worth keeping.
Because disk capacity is orders of magnitude larger than RAM, tiering shifts the question from “what can we afford to keep?” to “why would we discard saved work?” For a fleet serving thousands of long conversations, this means the system prompt, individual requests, and full conversation histories can remain warm somewhere in the cluster instead of being recomputed on each request.
An infographic titled "The Hierarchy Sorts Itself" showing a "Storage tier" grid of colored squares above three rounded icons labeled "the prompt," "every request," and "the history." It visually represents how items are organized into different storage tiers.
Routing and metadata. When blocks can exist on multiple floors across many pods, simply writing them to disk is not enough — the system must know where to find them. The KV index records not only that a block exists but also which pod currently owns it and which floor it’s on. That index lets the router:
  • Direct a block request to the pod that owns the block.
  • Let the owning pod promote the block the moment a conversation becomes active again.
This approach keeps requests local when possible and minimizes network hops and recomputation. In our example cluster every pod is CPU-only, so the machine’s main RAM is the fastest available tier on-premises. We therefore describe the full GPU → RAM → Disk hierarchy but demonstrate the tiering behavior even when some physical tiers (e.g., GPU) are absent.
An infographic titled "What Our Cluster Can Prove" showing a CPU-only cluster with three vLLM containers. Below it are stacked resource blocks labeled GPU, RAM, and Disk.
What this cluster demonstrates in practice is the core behavior: a long, growing conversation that returns repeatedly, with the KV cache absorbing previously seen tokens and preventing repeated prefill work. That avoids repeated compute and significantly improves latency for recurrent conversational workloads.
A dark-themed diagram titled "What Our Cluster Can Prove" showing a left "Conversation" box with stacked message bubbles connected by an arrow to a right "KV cache" box filled with small square cells. The graphic uses neon-like pink and teal outlines on a deep teal background.
When using a shared storage tier (e.g., PVC), ensure consistent access patterns and sufficient I/O capacity. Under-provisioned disk or high contention can become a bottleneck and negate the benefits of tiering.
Links and references Summary Tiered KV caching — GPU → RAM → Disk — lets model servers keep as much useful state warm as possible. By demoting rather than deleting blocks and recording ownership and floor metadata, a fleet can avoid repeated prefill work, reduce latency on recurring conversations, and scale to serve many long-lived interactions.

Watch Video