> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# The Speed Limit

> How GPU memory bandwidth limits transformer decode throughput and how prefill versus decode, KV caching, and incremental decoding optimize latency and token generation

Earlier we split a single inference request into two phases: the prompt-processing phase (prefill) and the token-generation phase (decode). Empirically, the bottleneck shifts between these phases: prefill is compute-bound, while decode is often memory-bound. Below we explain why reading memory caps decode speed, how that limit shows up in metrics, and how modern systems avoid redoing expensive work for every token.

Why does reading memory cap the speed?

GPUs have powerful compute units but only a modest amount of on-chip memory (registers and caches). Model weights for large transformers normally live in GPU VRAM. For each generated token the runtime must transfer the relevant slices of model weights from VRAM into the compute units, run the math, and then stream in the next slices. That streaming of weights happens roughly once per token for memory-resident models, so the GPU’s memory bandwidth sets a practical upper bound on token throughput.

![A stylized diagram of a GPU showing a grid of blue core blocks on the left and a labeled VRAM section on the right, with the caption "The Cores Have to Be Fed" above. The image uses a dark background and simple neon-blue and pink highlights.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/neon-gpu-cores-vram-diagram.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=344954e8ae2f99a33e90536a50a5beb2)

A high-end GPU might move on the order of 3 TB/s between VRAM and the compute units (exact numbers vary by architecture and memory type). For a 16 GB model, the simple bandwidth-bound estimate looks like:

```text theme={null}
3 TB/s ÷ 16 GB ≈ 200 full-model reads per second ≈ 200 tokens/sec
```

This converts into a per-token gap of about 5 ms between outputs — a metric commonly reported as Time Per Output Token (TPOT). The initial latency before any token appears is Time To First Token (TTFT). Benchmarks and SLAs usually report both TTFT and TPOT because together they summarize serving responsiveness.

![An infographic titled "The Two Numbers" with two circular gauges: a blue one on the left labeled "2 s" for time to first token, and a green one on the right labeled "5 ms" for time per output token.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/two-numbers-gauges-2s-5ms.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=31c0eedbddc507a201d2bd034255bb76)

Key definitions

| Metric | Meaning | Typical cause |
| - | - | - |
| TTFT (Time To First Token) | Latency before the first generated token arrives | Prefill compute and initialization |
| TPOT (Time Per Output Token) | Gap between successive generated tokens | Memory bandwidth and incremental work |

If you double the model size without increasing bandwidth, you halve the maximum token throughput — the compute units finish their work and then idle while waiting for the next weight slices. In short: decode throughput is often limited by GPU memory bandwidth rather than raw arithmetic throughput.

Two kinds of work inside one request

* Prefill: Process the prompt and compute its representations. This is a larger one-time compute burst and tends to be compute-limited.
* Decode: Generate tokens while repeatedly accessing model weights. This is a steady, repeated workload that tends to be memory-limited.

Most deployments run both phases on the same GPU, switching between them. That works at smaller scale but becomes inefficient as model sizes and concurrency increase.

![An infographic titled "Two Kinds of Work, One GPU" comparing Prefill (left) and Decode (right): Prefill processes the whole input in one pass ("the math is the limit"), while Decode does a separate one-pass step per token ("memory is the limit").](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/two-kinds-one-gpu-prefill-decode.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=fe521664f1326a4ccba1f4beb97d7a1d)

Context growth and why naive re-computation is infeasible

Every new token must condition on the entire conversation so far: the original prompt plus all previously generated tokens. If we naively re-run the full prefill pass for every new token, each token would incur the full TTFT — causing a long pause before every token and making interactive chat unusable. Concretely:

* Token 1: compute representations for prompt → generate token 1.
* Token 2: would need prompt + token 1 → naive approach would recompute everything.
* Token 3: would need prompt + tokens 1–2 → and so on.

![An infographic titled "What Each Word Costs" showing two dashed rows of rounded word tiles reading "Why does the first word take" and an arrow from the top row pointing to a glowing green box labeled "Because." It appears to illustrate how words or tokens in a prompt map to a single response token.](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/word-token-mapping-infographic.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=38e34e1493dc99c67e2199b6d678c01d)

If a system re-executed the full prefill on every token, users would perceive a fresh long pause before each word:

![A dark-themed interface titled "A Fresh Pause Before Every Word" showing four circular timers each marked "2 s" and two green rounded buttons labeled "Because" and "it."](https://mintcdn.com/kodekloud-c4ac6d9a/OJzUA92tPKBc1TNu/images/llm-d-A-Deep-Dive/Inside-a-Single-Request/The-Speed-Limit/fresh-pause-timers-2s-because-it.jpg?fit=max\&auto=format\&n=OJzUA92tPKBc1TNu\&q=85\&s=c6c1f2260a7ef0822a6a8c3d0dbefa90)

How interactive systems get fast: caching and incremental decoding

The practical solution is to preserve and reuse parts of the computation so that each new token requires only minimal incremental work. Common techniques include:

* KV caching (caching attention key/value tensors) so the model doesn’t recompute attention over the entire history every time.
* Incremental decoding that executes only the necessary layers/blocks for the new token slices.
* Beam- or cache-aware strategies that limit recomputation when exploring multiple hypotheses.

KV (key/value) caching and incremental decoding are essential optimizations for low-latency serving. They transform an O(length) recomputation per token into a near-constant extra cost, allowing the runtime to approach the limits set by memory bandwidth and token generation rates.

Practical implications and trade-offs

* Scaling model size without increasing memory bandwidth reduces TPOT proportionally.
* Increasing concurrency (many simultaneous streams) changes the effective bandwidth per stream and can raise TTFT if GPUs are shared.
* Caching accelerates serving but consumes memory; large KV caches can be a limiting factor for very long contexts.
* Architectures and runtimes that minimize VRAM transfers (fused kernels, weight streaming optimizations, and specialized memory hierarchies) can push TPOT closer to the hardware limits.

Further reading and references

* [Transformers and Attention Mechanisms](https://arxiv.org/abs/1706.03762)
* NVIDIA memory and performance guides (search for device-specific bandwidth numbers)
* Practical posts on KV caching and efficient generation from major ML infra projects (Hugging Face, Meta, NVIDIA)

By understanding the split between compute-limited prefill and memory-limited decode — and by applying caching and incremental strategies — serving systems keep chat interactive even as models grow.

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/343ab1ed-4576-43cf-96c6-cfaaebdcd6ef/lesson/d5eed7bc-de62-4f1d-91ac-48223dde62a3)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.