> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agentic Serving

> Optimizing serving infrastructure for agentic model workloads by caching shared context, maintaining program affinity, precise routing, tiered caches, and per-flow fairness.

In earlier lessons, the "user" on the other end of each request was usually a person — someone who types a question, reads an answer, thinks, and replies. Production workloads today increasingly look different: the "user" is often another program, an automated agent.

An agent is a model running in a loop with tools: it receives a task, chooses a tool, reads the tool's output, decides the next action, picks another tool, and repeats — sometimes dozens of times before a human sees the final result. A coding agent, for example, may read files, run tests, search a repository, and propose patches.

![A diagram titled "A Model in a Loop" showing a coding agent inside a dashed box interacting with tools and a model in a repeated feedback loop (×24). The model's output is then sent to a user icon on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/model-loop-coding-agent-tools-user.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=69080025b5b25df8c5d8a6d1ca9a3fe9)

Every step in that loop is a separate model call. That changes the unit you need to optimize for: you are no longer serving a single request, but a chained program of many requests. This flips the latency math: an interactive human measures wait time as the finish time of the entire job, not the latency of one call.

![A dark slide titled "A Program of Calls" shows a dashed box labeled "Program" containing a row of connected pink rounded squares with small robot icons. The chain of calls points to a separate purple user/person icon on the right.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/program-call-chain-robots-to-user.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=35714e9b3332249693f1fcf98bee1d0c)

Because each step depends on prior steps, the time-to-first-token (TTFT) becomes a repeated toll booth the whole program must pass through. Reduce the pause at that point and the entire program accelerates; ignore it and the tolls compound across turns.

Look inside one of those calls and you’ll see an extreme shape that defines the optimization opportunity. Each call typically contains:

* A large shared prefix: the system prompt, the catalog of available tools, and the accumulated history of previous turns.
* A tiny new tail: the most recent tool output or the new instruction appended at the end.

Decoding for many turns is therefore light: often only enough work to select the next tool.

Published traces from real coding agents show this clearly: by turn 30 an agent’s context can approach \~80,000 tokens, while each new turn tends to add only a few hundred to a few thousand tokens.

![A dark-themed bar chart titled "The Published Numbers" showing steadily increasing teal bars from turn 1 to turn 30, reaching about 80,000 tokens and labeled "Agent traces." Each bar has a small blue segment on top indicating additional counts (hundreds to thousands).](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/published-numbers-agent-traces-bar-chart.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=dff336e0aad2ac5844fef604890c678c)

Across corpora, input has outweighed output by roughly `130:1`. In practice this means an agent’s workload is dominated by rereading what it already knows. That makes caching not merely an optimization but the load-bearing wall of a production agent platform.

Cache the shared prefix (system prompt, tool catalog, and history), and the incremental cost of each next turn collapses to just the delta — the new tool output appended at the tail. Miss that cache and fleets will repeatedly resend 80,000 tokens to append an 800-token result.

![An infographic titled "The Load-Bearing Wall" showing two pod diagrams each linked to a horizontal cost bar. The top row has a green check and a shorter teal cost bar with a small command icon, while the bottom row has a red X and a much longer blue cost bar labeled "80,000."](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/load-bearing-wall-pod-costs-80000.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=5516e330868539881faf6f5890589df6)

Caching the shared prefix (system prompt, tool catalog, and history) is essential for agent workloads. Without it, you repeatedly pay the full context cost for each turn.

Because cross-turn reuse is systematic for agents, the platform needs to treat program state as sticky. Cache-aware routing and affinity mechanisms (for example, weighted-trie scoring) should keep an executing agent returning to the pod that stores its life story.

![A schematic titled "Program Affinity" showing a service architecture with a Gateway and EPP component routing to multiple pods (pod 0, pod 1, pod 2). Icons and labels like "llm-d", "Cache-aware", "Tiered cache", and "Fairness" annotate caching and affinity features.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/program-affinity-gateway-epp-pods-cacheaware.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=3e96e0019df328b152c7bf3afea32b36)

Precise routing matters: a single stale routing decision that adds one slow turn for a chat user can slow an entire agent program for every remaining step.

![An infographic titled "Precise Routing" showing an llm-d routing and caching workflow with icons labeled "Cache-aware", "Precise", "Tiered cache", and "Fairness." A row of request icons forms a timeline that highlights a "Stale guess" point and a "One slow turn" indicating latency.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/precise-routing-llmd-cache-stale-latency.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=74dbcf475ec30efcdaeb916b6eb27e6f)

A tiered cache is indispensable for another reason: tool calls can take seconds or longer (for example, running tests). While an agent waits for external work, its cached blocks sit idle and are prime eviction candidates — exactly when you know they will be needed again. Load balancers and routing must therefore maintain session affinity across those gaps.

Consider a simple tool-driven pause common in coding agents:

```bash theme={null}
$ pytest
```

If the platform evicts that agent’s cached prefix while tests are running, the system will pay to rehydrate the full context when the agent resumes. To prevent this you need:

* Cache-aware routing (keep the run on the same pod that already stores its prefix)
* Tiered caches (fast local cache + larger remote cache to avoid eviction)
* Fairness controls (prevent noisy agents from starving interactive users)

An agent platform is inherently a noisy-neighbor environment: one user action can fan out into dozens of model calls. Per-flow fairness and flow control let agent traffic and human traffic share the same fleet without letting agents monopolize capacity and slow interactive humans.

![A dark-themed architecture diagram titled "Per-Flow Fairness" showing icons for features (cache-aware, precise, tiered cache, fairness) and a data flow from a user through a Gateway to an EPP component which distributes requests to multiple pods (pod 0, pod 1, pod 2). The llm-d logo appears at the top left.](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/per-flow-fairness-gateway-epp-pods.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=c861e49e535335d2e0e953cfdee3b2a3)

All of the machinery you’ve learned so far — cache-aware routing, tiered caches, precise routing, and per-flow fairness — doesn’t just survive the agent era; it is the foundation the agent era runs on.

Below is a concise summary of the core platform features and why they are essential for agentic serving.

| Feature | Why it matters | Implementation notes |
| - | - | - |
| `Cache-aware routing` | Keeps an executing agent returning to the pod with its cached prefix, saving massive re-reads | Use affinity keys based on program ID and weighted-trie scoring |
| `Tiered cache` | Prevents eviction during external waits (e.g., tests), provides fast local hits and resilient fallbacks | Local in-memory cache + remote distributed cache (LRU with pinning) |
| `Precise routing` | Minimizes stale-guess penalties that compound across turns | Fast routing table updates and low-latency route recalculation |
| `Per-flow fairness` | Stops noisy agents from monopolizing model capacity and degrading interactive user experience | Rate limits, queueing policies, fair-share schedulers |

Finally, remember that text-only agents are just the beginning. Multimodal agents and longer-lived programs are arriving fast, and the same core principles — caching shared state, maintaining program affinity, precise routing, tiered caches, and fairness — apply across modalities and lifetimes.

![A dark-themed diagram titled "The Agent Era" showing a long green bar with repeated sync/refresh icons. Below it are labeled cards for "Gateway", "EPP", and three pods ("pod 0", "pod 1", "pod 2").](https://mintcdn.com/kodekloud-c4ac6d9a/uibir82za_XVXFvX/images/llm-d-A-Deep-Dive/Wrap-Up-Agents-and-Beyond/Agentic-Serving/agent-era-gateway-epp-pods.jpg?fit=max\&auto=format\&n=uibir82za_XVXFvX\&q=85\&s=4de903cb5ef738a6bf67ef07454e8771)

Links and references

* [Kubernetes Documentation](https://kubernetes.io/docs/)
* [PyTest Documentation](https://docs.pytest.org/)
* [Cache (computing) — Wikipedia](https://en.wikipedia.org/wiki/Cache_\(computing\))
* [Designing Low-Latency Systems — concepts on session affinity and routing](https://en.wikipedia.org/wiki/Load_balancing_\(computing\))

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/25c38cf7-ca62-4f39-9a00-e607af3b4dfe/lesson/ea0ad343-d74e-44d3-a78c-dde73a5e153c)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.