Skip to main content
This section explains how modern chat models avoid redoing expensive work for every generated word. Understanding the prefill phase, the KV cache, and prefix caching clarifies why chat systems are fast and how servers can reuse work across turns and users.

​
Prefill, KV cache, and decode — the core loop

When a request arrives the model performs a costly initial pass called the prefill. During prefill the model reads the entire prompt and computes intermediate activations for every token. That work is not thrown away — the model stores it in GPU memory as the KV cache.
  • Prefill: single, heavy pass over the incoming prompt that fills the KV cache.
  • KV cache: saved intermediate activations in GPU memory representing the history seen so far.
  • Decode phase: the model generates the answer token-by-token and reuses the KV cache so it doesn’t rerun prefill for every new token.
A stylized GPU diagram showing a grid of compute blocks on the left with some highlighted in blue and a large labeled VRAM block on the right. The image is titled "Keep It Instead."
Because the KV cache preserves the prefill work, each new token only requires the incremental computation for that token instead of reprocessing the entire preceding text. Without the KV cache, every token would force a full re-run of prefill across the entire history, which would make token generation prohibitively slow.
A slide titled "With It, and Without It" comparing token processing: the left side (without the cache) shows every word marked "redo," while the right side (with the cache) shows most words marked "saved" and only the final token as "new."
The prefill+KV cache pattern is why generation time per-token is small after the first output token. Prefill does the heavy lifting once; the KV cache lets decoding reuse that work.

​
One message, then gone — why each request forgets

The KV cache is usually freed when a reply completes. That means the server does not persist computed state across requests by default (GPU memory is valuable and must be recycled). So each new message usually starts fresh and must be re-prefilled from the beginning of the conversation.
  • Turn 1: the server reads your question and fills the cache.
  • Turn 2: the server re-reads turn 1 (question + answer) plus your new message and refills the cache.
  • Turn 3: the server re-reads all previous turns and the new message again.
A dark-themed diagram titled "One Message, Then Gone" showing a prompt icon on the left feeding into a central request box (keyboard/grid) that produces a reply icon on the right.
Because each request re-reads the entire conversation history, latency and computation increase rapidly as the conversation grows. In practice this causes per-request cost and latency to grow much faster than linearly with conversation length.

​
Why prefix caching matters

Prefix caching (also called prompt caching or cached prefixes) avoids re-reading the same beginning of a conversation on every request. Instead of freeing the KV cache after each reply, the server stores those cached KV blocks indexed by the text that produced them. When a later request arrives with the same beginning (the same prefix), the server reuses the cached state for that prefix and only prefill-processes the new trailing tokens.
A dark-blue UI screen titled "Only the New Part" showing three message boxes. The first two messages have green checkmarks ("Why does the first word take longest?" and "Because it reads it all."), while the third ("And what about the second word?") has a red X.
With prefix caching the cost per new turn is mostly the work to prefill the new tokens; the rest is supplied from the cached prefix, so per-turn latency stays roughly flat regardless of how long the cached prefix is.
Many hosted APIs already offer prompt or prefix caching. For example, Anthropic’s Claude API uses prompt caching to reduce cost; a cached token often costs roughly a tenth of a fresh prefill token.
An infographic titled "You May Have Paid for This" showing an API icon and the phrase "prompt caching." It compares two progress bars: a long blue bar labeled "a fresh token — full price" and a much shorter green bar labeled "a cached token — about a tenth."

​
Prefix caching across users — shared beginnings

A powerful property of cached computation is that it depends only on the text, not the user. If many conversations share the same initial instructions or system prompt, the server can compute that prefix once and reuse the cached KV blocks for all users who start the same way. From the model’s standpoint this is identical to reusing a prefix for subsequent turns in a single conversation.
A dark slide titled "One Opening, Everybody" showing a green square labeled "Assistant" with a person icon and an arrow pointing to a glowing blue square labeled "Instructions."

​
What an inference server already does

An inference server typically performs these responsibilities:
  • Hold the model weights in GPU memory.
  • Read the incoming prompt in a prefill burst.
  • Generate the answer token-by-token during decode while reusing the KV cache.
  • Optionally keep the KV cache between requests (prefix caching) so later requests can reuse work.
A dark presentation slide titled "What One Server Already Does" showing five rounded cards with icons labeled: "Holds the model," "Reads your prompt," "Writes the answer," "Saves that work," and "Keeps it between turns" (highlighted). The layout emphasizes the last card with a glow.
When the same cached beginning is reused by many users, the server can hand out identical cached prefixes to all of them. That enables a single GPU to serve many concurrent users much more efficiently than re-computing the same prefix independently for every user.
An infographic titled "That Last Part Is the Tell" showing a GPU illustration with a VRAM block on the left and a grid of user icons labeled "Everyone" on the right.

​
Practical implications and next questions

  • Memory trade-offs: caching prefixes reduces compute but requires reserved GPU memory to store KV blocks.
  • Shared prefixes increase throughput: identical starts across many users multiply the value of each cached prefix.
  • Capacity planning: how many users can a single GPU serve depends on model size, KV cache size per token, and how many distinct prefixes must be stored.
Prefix caching lowers compute but consumes GPU memory. Servers must balance how many cached prefixes they keep versus memory needed for active inference work. Eviction policies and prefix deduplication are essential in production.
If you’d like, the next section can dive into numeric examples (KV memory per token, approximate user counts for given GPUs), eviction strategies, or how to implement prefix caching in an inference stack.

Watch Video