Skip to main content
In the previous lesson we saw that a large language model (LLM) generates text one token at a time in a loop. That loop, however, contains two very different phases with distinct performance characteristics. Understanding the split between those phases explains why serving large models is often surprising: there’s a noticeable initial pause, then a steady stream of tokens. Think of a familiar interaction: you paste a long block of text into ChatGPT and press Enter. After a short pause nothing appears, then the reply begins streaming out steadily. Those two moments correspond to two phases: the pause (prefill) and the stream (decode). Below we describe what happens in each phase and why their bottlenecks differ.

​
Phase 1 — Prefill: the model reads the entire prompt first

Before the model emits a single output token, it must process your entire prompt — every token you provided — in a single forward pass. All input tokens are read together; nothing is gradually “filled in” during this step.
A slide titled "It Reads All of It First" showing a teal text box with a sample sentence split into blue token blocks. Arrows from the tokens lead into a pink rounded square labeled "the model" with a small robot icon.
What happens during this pass:
  • The model weights and architecture are used to compute activations over the full input.
  • Thousands of GPU cores (or equivalent accelerators) work in parallel to produce the intermediate state and the first output token.
  • This single, highly-parallel compute burst determines how long you wait before seeing any output.
This initial phase is called prefill. The observed wait time is commonly measured as Time to First Token (TTFT). Short prompts (e.g., “What’s the capital of France?”) are only a few tokens, so the prefill pass completes quickly and TTFT is small. Very long inputs (a multi-thousand-token contract or long document) require a much larger pass and create a noticeable pause before streaming begins.
The initial pause you observe when a model starts a response is mostly the prefill step. If you need to reduce perceived latency, reduce the context size or split input into smaller chunks so that the forward pass is faster and TTFT drops.

​
Phase 2 — Decode: the model writes the answer token-by-token

After prefill produces the first token, the model enters the decode loop: generating tokens one at a time. Each new token requires another pass through parts of the model, but this phase is fundamentally different in what limits performance.
A dark-themed slide titled "Phase 2: Writing the Answer" showing a ChatGPT-style chat window with the user message "Why does the first word take longest?" and the reply "Because it reads it all." To the right is a simple diagram featuring a pink model icon labeled "the model" and a "pass 5" indicator.
Key points about decode:
  • The decoder uses the growing key/value cache (the KV cache) to store intermediate attention states from earlier tokens for efficient autoregressive decoding.
  • For each new token, the GPU must repeatedly read model state (weights and the KV cache) from VRAM.
  • Because the loop is iterative, many memory reads are repeated: generating N tokens involves repeated access to large model state for each iteration.
The result is that decode is usually memory-bandwidth-limited rather than compute-limited. The GPU cores spend a lot of time waiting for data to be fetched from VRAM, and each memory load typically yields just one more output token. Thus the speed of memory transfers (memory bandwidth) defines the steady streaming rate you observe.
A technical diagram titled "The Cores Are Barely Working" showing a GPU with VRAM and a KV cache, illustrating seven loads / seven tokens. Green squares at the bottom represent the tokens being processed by the GPU.
Decode is typically memory-bound: adding more compute cores alone won’t improve streaming rate if the GPU’s memory bandwidth is the bottleneck. For faster streaming, look to architectures or hardware with higher VRAM bandwidth or optimizations that reduce repeated memory reads (e.g., more compact KV representations or caching strategies).

​
Quick comparison

​
Practical takeaways

  • The initial pause you notice is dominated by prefill and scales with the prompt size; keep context sizes reasonable when low latency is important.
  • The steady streaming speed is set by the decode phase and often limited by GPU memory bandwidth and KV cache behavior.
  • Optimizations for lower latency will differ depending on which phase dominates: reduce context length to improve TTFT; reduce memory access per token or increase memory bandwidth to improve steady-state throughput.
Further reading and references:
  • ChatGPT — example of the familiar pause-then-stream behavior
  • Memory bandwidth and GPU architecture discussions (search hardware vendor docs for specifics on VRAM and bandwidth trade-offs)

Watch Video