Explains how autoregressive language models tokenize text, perform initial context encoding and per-token generation, and how token counts affect latency, cost, and serving complexity
Last time we left off with a real server running and one question still unanswered: what happens between your prompt arriving and the model returning an answer? Opening that up reveals the core operational challenges of serving large language models (LLMs).Before we go further, one key word: a language model doesn’t read and write whole words like we do. It breaks text into pieces called tokens. A token is roughly three-quarters of a word on average for English, but exact token boundaries depend on the model’s tokenizer (byte-pair encoding, SentencePiece, or related schemes), so sometimes a token is a whole word and sometimes only part of one.Take a short sentence like this:Serving LLMs is not like serving web apps.That’s eight words, but it comes to nine tokens, and
everything in the request counts as tokens — what the model can read (context), what you pay, and how long it takes. Keep that one word in mind.The core behavior for autoregressive LLMs is simple but powerful: given all the text so far, the model predicts the single next token. It appends that predicted token to the context, and then predicts the next token. It repeats this loop until it stops.So a 100-token answer isn’t computed in one giant pass; it’s the model running that single-token prediction loop 100 times. That’s also why you see answers “type out” token by token in streaming interfaces.
> What's the capital of France?Paris is the capital of France, and its largest city, with about 2.1 millionresidents.
Because tokens are produced incrementally, streaming output as each token is generated is straightforward: each produced token can be sent to the client with very little additional computation beyond the generation itself. Two practical consequences follow, and both cause operational stress later.The first consequence is latency: these requests take a long time. A normal web request often completes in milliseconds, but an answer with hundreds of tokens requires hundreds of loop iterations, which can tie up expensive GPU resources for seconds.
The second, and larger, consequence is variability: requests come in wildly different sizes. “What’s the capital of France?” is tiny. “Summarize this 100-page contract” might be tens of thousands of tokens in and thousands out. Both hit the same API endpoint, but one can require orders of magnitude more compute, memory, and time than the other.
Both long runtimes and high variability matter because they stress the server in different ways. Understanding the internal loop helps explain why.Two halves of the generation loopAlthough the high-level loop is simple, each iteration actually relies on two different phases that stress hardware differently:
Context encoding (initial pass): the model runs a large forward pass over the entire input context to compute internal key/value (KV) caches for the attention layers. This step uses large matrix multiplications across the full prompt and is heavy on compute and memory bandwidth. Very long contexts increase the memory footprint substantially.
Per-token generation (incremental decode): for each new token the model runs a forward step that uses the cached keys and values to produce the next-token distribution, samples or selects a token, and appends it to the context. Because of caching, this per-token step is cheaper than reprocessing the whole prompt, but it still attends to all prior tokens (so cost grows with sequence length) and involves many small GPU kernel launches. Per-token work is latency-sensitive and often less throughput-efficient on GPUs than the big batched operations of the initial pass.
This split—an expensive upfront context encoding followed by many smaller per-token steps—explains a common user experience: a short pause after you submit a prompt, then streaming tokens begin. That pause is the model computing the heavy initial pass (building the KV cache); once it finishes, the model decodes tokens one by one and streams them out.Quick comparison
Phase
What happens
Compute & memory profile
Impact
Context encoding (initial pass)
One large forward pass over the entire prompt to build KV caches
Heavy matrix multiplications, high memory bandwidth, large memory footprint for long contexts
Initial latency before any tokens stream; can be the dominant GPU cost for very long prompts
Per-token generation (incremental decode)
Repeated smaller forward steps using cached KV pairs to produce the next token
Smaller compute per step but many iterations; many small GPU kernels; cost grows with decoded length
Streaming possible with low per-token overhead but overall latency accumulates; less GPU-efficient for many small steps
Very long contexts and large generated outputs can cause high costs, increased latency, or out-of-memory errors on GPUs. Design prompts and client expectations accordingly.
Why this matters for serving LLMs
Token count and context length directly affect cost, latency, and memory. Short prompts are fast and cheap; long contexts increase memory use and per-token compute cost.
Variability makes capacity planning and fair scheduling difficult: a single long request can monopolize resources and increase tail latency for others.
Streaming mitigates perceived latency but does not eliminate total compute time; it simply delivers tokens as they are produced.
Token count, context length, and how you structure prompts directly affect cost, latency, and memory. Short prompts may be fast and cheap; long contexts increase memory use and per-token cost.
