Optimizing serving infrastructure for agentic model workloads by caching shared context, maintaining program affinity, precise routing, tiered caches, and per-flow fairness.
In earlier lessons, the “user” on the other end of each request was usually a person — someone who types a question, reads an answer, thinks, and replies. Production workloads today increasingly look different: the “user” is often another program, an automated agent.An agent is a model running in a loop with tools: it receives a task, chooses a tool, reads the tool’s output, decides the next action, picks another tool, and repeats — sometimes dozens of times before a human sees the final result. A coding agent, for example, may read files, run tests, search a repository, and propose patches.
Every step in that loop is a separate model call. That changes the unit you need to optimize for: you are no longer serving a single request, but a chained program of many requests. This flips the latency math: an interactive human measures wait time as the finish time of the entire job, not the latency of one call.
Because each step depends on prior steps, the time-to-first-token (TTFT) becomes a repeated toll booth the whole program must pass through. Reduce the pause at that point and the entire program accelerates; ignore it and the tolls compound across turns.Look inside one of those calls and you’ll see an extreme shape that defines the optimization opportunity. Each call typically contains:
A large shared prefix: the system prompt, the catalog of available tools, and the accumulated history of previous turns.
A tiny new tail: the most recent tool output or the new instruction appended at the end.
Decoding for many turns is therefore light: often only enough work to select the next tool.Published traces from real coding agents show this clearly: by turn 30 an agent’s context can approach ~80,000 tokens, while each new turn tends to add only a few hundred to a few thousand tokens.
Across corpora, input has outweighed output by roughly 130:1. In practice this means an agent’s workload is dominated by rereading what it already knows. That makes caching not merely an optimization but the load-bearing wall of a production agent platform.Cache the shared prefix (system prompt, tool catalog, and history), and the incremental cost of each next turn collapses to just the delta — the new tool output appended at the tail. Miss that cache and fleets will repeatedly resend 80,000 tokens to append an 800-token result.
Caching the shared prefix (system prompt, tool catalog, and history) is essential for agent workloads. Without it, you repeatedly pay the full context cost for each turn.
Because cross-turn reuse is systematic for agents, the platform needs to treat program state as sticky. Cache-aware routing and affinity mechanisms (for example, weighted-trie scoring) should keep an executing agent returning to the pod that stores its life story.
Precise routing matters: a single stale routing decision that adds one slow turn for a chat user can slow an entire agent program for every remaining step.
A tiered cache is indispensable for another reason: tool calls can take seconds or longer (for example, running tests). While an agent waits for external work, its cached blocks sit idle and are prime eviction candidates — exactly when you know they will be needed again. Load balancers and routing must therefore maintain session affinity across those gaps.Consider a simple tool-driven pause common in coding agents:
$ pytest
If the platform evicts that agent’s cached prefix while tests are running, the system will pay to rehydrate the full context when the agent resumes. To prevent this you need:
Cache-aware routing (keep the run on the same pod that already stores its prefix)
Tiered caches (fast local cache + larger remote cache to avoid eviction)
Fairness controls (prevent noisy agents from starving interactive users)
An agent platform is inherently a noisy-neighbor environment: one user action can fan out into dozens of model calls. Per-flow fairness and flow control let agent traffic and human traffic share the same fleet without letting agents monopolize capacity and slow interactive humans.
All of the machinery you’ve learned so far — cache-aware routing, tiered caches, precise routing, and per-flow fairness — doesn’t just survive the agent era; it is the foundation the agent era runs on.Below is a concise summary of the core platform features and why they are essential for agentic serving.
Feature
Why it matters
Implementation notes
Cache-aware routing
Keeps an executing agent returning to the pod with its cached prefix, saving massive re-reads
Use affinity keys based on program ID and weighted-trie scoring
Tiered cache
Prevents eviction during external waits (e.g., tests), provides fast local hits and resilient fallbacks
Local in-memory cache + remote distributed cache (LRU with pinning)
Precise routing
Minimizes stale-guess penalties that compound across turns
Fast routing table updates and low-latency route recalculation
Per-flow fairness
Stops noisy agents from monopolizing model capacity and degrading interactive user experience
Finally, remember that text-only agents are just the beginning. Multimodal agents and longer-lived programs are arriving fast, and the same core principles — caching shared state, maintaining program affinity, precise routing, tiered caches, and fairness — apply across modalities and lifetimes.
