> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# When the Scratchpad Overflows

> Tiered caching and live block indexing that move LLM conversation state between GPU RAM and disk to avoid recomputation and enable reliable routing.

The last session ended on an uneasy note.

All of that clever routing depends on one assumption: the saved work — the KV cache — is still available when you return. On a single machine, that saved work lives in one place:

![An infographic titled "It All Rests on One Thing" shows a purple box labeled "All that clever routing" connected down to a teal box labeled "Your saved work — the KV cache." A small purple question‑mark icon sits to the right on a dark background.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/kv-cache-clever-routing-infographic-questionmark.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=5f14fe3424aa0109b36fc0a319af5817)

That one place is the GPU’s memory — the same place where space is most desperately short.

What happens when that memory fills up? To answer, picture the GPU’s internal layout: the model’s weights take a fixed share of memory, and everything left over stores the scratchpads — the per-conversation cached “saved work.” That layout is repeated for every conversation on the GPU. As load grows, the leftover space shrinks. When a new request needs room, the server must pick something to discard. The practical choice is eviction of the item that has gone the longest without being used.

![A stylized diagram titled "And Then It Fills" showing GPU memory occupied by a large "Model" block and several server-like modules until an orange X marks "no room." A caption reads "The server gets busy."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/gpu-memory-model-server-no-room.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=5590d770a0108205ccf723fdf7b1a830)

This eviction policy is called Least Recently Used (LRU).

![A dark UI mockup titled "Least Recently Used" showing a row of rounded square tiles labeled with times since last touched (8 s, 22 s, 1 m, 45 s, 4 m — highlighted, 12 s, 2 m). A stopwatch-style icon on the left is labeled "quiet the longest."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/least-recently-used-tiles-stopwatch.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=2fb7b2017de407545a4f848d639eca97)

Whoever has been quiet the longest loses their seat. Often that “who” is your own conversation — the one from a few minutes ago you were about to continue. The GPU silently discards that conversation’s scratchpad blocks; there’s no error or notice. The next time you send a message, the server must recompute the prefill across the entire conversation and rebuild those blocks — precisely the costly work the cache was intended to avoid.

This behaviour is not a bug. With one limited, high-speed tier for saved work, a server has only two options: evict something or refuse new requests. Refusing requests is not generally acceptable, so eviction is the unavoidable practical choice. What eviction discards are just blocks of numbers (tensors) sitting in the wrong place — not broken work or incomplete answers.

Computers already provide multiple kinds of memory. The GPU is the penthouse: blistering fast but very small. One floor down is the machine’s RAM — slower but far roomier. Beneath that is disk — slower still, but practically limitless.

Because the saved work is just tensors, moving a block down one floor (GPU → RAM → disk) and later pulling it back up costs significantly less than recomputing it (the prefill pass). Creating the block is the expensive part; moving bytes between tiers is cheap by comparison.

To exploit that difference, [llm-d: A Deep Dive](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive) implements a tiered cache. When the GPU fills, blocks are demoted rather than destroyed:

* If the GPU is full, blocks are moved down to RAM.
* If RAM fills, blocks are moved down to disk.
* When a conversation becomes active again, its blocks get promoted back up to the tier they need.

Each tier holds the data it’s best suited for: the GPU keeps the white-hot working set, RAM holds recently active conversations, and disk stores the long tail that might still return.

![An infographic titled "Each Floor, What It's Suited For" showing three memory/storage tiers: GPU (fastest), RAM (slower), and Disk (slowest). Labels like "White-hot working set" and "Recently active" indicate which data belongs on each layer.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/memory-tiers-gpu-ram-disk.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=18967bd20de2ed6d2d22510805814375)

A quick reference:

| Tier | Typical role | Performance tradeoff |
| - | - | - |
| GPU | White-hot working set (active responses) | Fastest execution, smallest capacity |
| RAM | Recently active conversations | Slower than GPU but much larger; good for short-term retention |
| Disk | Long tail / seldom-used conversations | Slowest, essentially unlimited; still much faster than recomputing prefill for long contexts |

Here’s the payoff: even loading blocks from the slowest tier (disk) and copying them back into GPU memory is typically much faster than redoing the entire prefill computation. The longer the conversation, the larger the savings. Measurements from the [llm-d: A Deep Dive](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive) project show long prompts restored from storage can be up to nearly 17× faster than recomputing the prefill.

![Infographic titled "Nearly 17 Times Faster" comparing a short green "Reading back" bar labeled fast with a long purple "Recomputing" bar labeled slow. It cites a 17x speed improvement measured by the llm-d project and includes icons and a KodeKloud copyright.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/reading-back-recomputing-17x-speed.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=4a34b83335280bf41d5b8acc80aa3c20)

Because blocks move between tiers and occasionally expire entirely, the location of any conversation’s saved work is constantly changing. A router that relied on stale information could send you to a pod that just finished forgetting you. To prevent that, the model servers publish every state change.

Every time a block is created, moved, or evicted, the pod publishes an event. An [llm-d: A Deep Dive](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive) index maintains a live map of which blocks exist, which pod holds them, and on which tier (GPU, RAM, or disk) they currently reside.

![A dark-themed diagram titled "The Pods Publish Every Move" showing a vLLM pod publishing events (created, moved, evicted) that update a live block index. The index lists blocks (e.g., blk-7f2, blk-3a9, blk-c14) mapped to pods and storage types (GPU, RAM, disk).](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/vllm-pod-publishes-live-block-index.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=e53d98c470c2d65c72164175eecb0610)

The router computes routing scores based on what the live index reports as actually present, not on historical assumptions. That keeps routing accurate as blocks migrate.

![An infographic titled "Scored on What's Actually There" showing a live block index on the left mapping block IDs to pods and resource types. On the right are three pods (A, B, C) with TTFT and TPOT latency numbers, with pod C highlighted as the fastest.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/block-index-pods-latency-fastest-c.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=5d454f57e275210c0f96b2fa21dfc895)

Putting the pieces together:

* A model is a large file of numbers that needs a GPU to execute.
* Generation proceeds token-by-token in two phases: the prefill (conditioning on history) and the decode (autoregressive steps).
* The scratchpad makes streaming feel instant. Batching keeps it affordable. Sharding makes very large models feasible.
* In a fleet, routing becomes complex. [llm-d: A Deep Dive](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive) solves this with a router that knows where saved work currently lives and a tiered cache that survives temporary space shortages.

![A dark-themed slide titled "Look at It Stacked Up" showing a vertical stack of labeled bars. Each bar lists short points about how large AI models operate and scale (GPU-backed model files, token-by-token generation, streaming/pause, scratchpad, batching, sharding, and routing/fleet).](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/ai-model-scaling-stacked-bars.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=fcddd7f5c322427af8482106def20706)

Every piece of this design is a response to a constraint created by the piece before it. Next up is demonstration: a real GPU cluster, real pods, and live requests so you can observe how the system behaves in practice.

![A dark-themed diagram titled "Let's Go Look" showing a "decode pool" with three pods (pod 0, pod 1, pod 2) each with progress bars and queue numbers, and a purple "Gateway" box labeled "Running live." A Kubernetes icon sits at the left, indicating the pods are managed in a cluster.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Meet-llm-d/When-the-Scratchpad-Overflows/lets-go-look-decode-pool-gateway.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=02e545359406c80bfa420197b58be74b)

Tiered caching (GPU → RAM → disk) plus a live block index delivers both strong performance and predictable routing: copying blocks between tiers is cheap compared with recomputing them, and the router always knows where to send requests.

Further reading and references

* [llm-d: A Deep Dive — course page](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive)
* Kubernetes concepts: [Kubernetes Basics](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
* Practical caching strategies and memory tiers: see system architecture and cache design guides for GPU-backed inference systems.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/ee1bfe23-198d-4b8d-a23a-e43b6730da53/lesson/19309d9a-a6d6-4235-b886-f005e23ddcfc)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.