Demo An Agent Style Multi Turn Workload

One demo left: the two halves of this course shaking hands. We take the agent workload described earlier and run it on the fleet you spent six modules learning to operate. In this lesson we act as a coding agent across eight turns and observe how a program-scale cache behaves — doing for code what a scratchpad did for conversation. Every real agent session begins with a brief: who you are, which tools exist, the rules, the task, and repository background. Our example prompt is fictional but plausibly shaped. Note its size — this prompt alone is larger than a full cache block.

This session prompt defines available tools and strict rules: each step the agent must choose exactly one tool, explain the choice in one sentence, and stop after proposing the tool call. This constraint forces short, focused tool interactions and simulates an agent operating in a CI environment.

Session prompt (multiline for readability):

# Session prompt (multiline for readability)
SESSION=$(cat <<'EOF'
You are a coding agent operating inside a continuous integration system for a platform team. 
You have access to the following tools. read_file returns the contents of a file at a path. 
list_directory returns the names of files in a directory. run_tests runs the test suite for a package 
and returns failures. search_code finds lines matching a pattern across the repository. write_patch 
applies a unified diff to a file. git_log returns recent commit messages. 

Rules for every step: choose exactly one tool, explain the choice in one sentence, and stop after 
proposing the tool call. The current task is to find and fix a failing unit test in the billing 
package without changing any public interface. Repository background: the service is written in Go, 
uses the standard testing package, stores fixtures under testdata, and follows a layout with cmd, 
internal, and pkg directories. The previous engineer noted flaky behavior around time zone handling 
in invoice date calculations.
EOF
)

Multi-turn loop Each lap of the loop appends a tool result to SESSION, asks the model for the next action, and records prompt token counts and elapsed time. The shell loop below runs eight turns. It mimics a real agent session where tool outputs accumulate in the prompt and make the model’s context grow.

for i in $(seq 1 8); do
  SESSION="${SESSION} Tool result ${i}: run_tests reported one failure in the billing package. The failing test is TestInvoiceDates in internal/billing/invoice_test.go, expected 2024-03-01T00:00:00Z but got 2024-02-29T23:00:00Z, a one hour offset. list_directory shows cmd, internal, pkg, testdata, go.mod, go.sum, and Makefile. search_code found three matching lines: a call to time.Date in invoice.go line 41, a location load of UTC in clock.go line 12, and a fixture timestamp in testdata/invoices.json line 7. git_log shows a commit two weeks ago titled "switch invoice dates to local time". The step budget allows ${i} more actions. Based on this result, choose the next single step.
Decision:"

  PAYLOAD=$(jq -n --arg prompt "${SESSION}" \
    '{model: "Qwen/Qwen2.5-0.5B-Instruct", prompt: $prompt, max_tokens: 30}')
  SECS=$(curl -s -o /tmp/turn.json -w '%{time_total}' \
    http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "${PAYLOAD}")
  TOKENS=$(jq -r '.usage.prompt_tokens' /tmp/turn.json)
  STEP=$(jq -r '.choices[0].text' /tmp/turn.json | tr '\n' ' ')
  echo "turn ${i}: prompt tokens ${TOKENS}, took ${SECS}s"
  SESSION="${SESSION} ${STEP}"
done

Why context balloons The loop’s additions are what make agent context balloon: each turn adds tool results that contain file paths, test logs, and other real-world noise. Typically, each tool result contributes many more tokens than the model emits in its decision, so the prompt grows fast while the model’s output remains compact. Example observed run (prompt token growth and elapsed wall time):

turn 1: prompt tokens 372, took 3.348598s
turn 2: prompt tokens 587, took 3.004413s
turn 3: prompt tokens 804, took 4.713803s
turn 4: prompt tokens 1020, took 4.954897s
turn 5: prompt tokens 1236, took 3.601795s
turn 6: prompt tokens 1452, took 4.145017s
turn 7: prompt tokens 1668, took 4.292234s
turn 8: prompt tokens 1884, took 3.075746s

For quick reference, here’s the same data as a table (useful for SEO and readability):

Turn	Prompt tokens	Elapsed time (s)
1	372	3.348598
2	587	3.004413
3	804	4.713803
4	1020	4.954897
5	1236	3.601795
6	1452	4.145017
7	1668	4.292234
8	1884	3.075746

Key observations

Tool outputs are verbose and typically account for most of the token growth. In this example, each tool result often contributes roughly six times the tokens the model itself generates.
Prompt tokens grow substantially across turns while wall-clock elapsed seconds per turn stay relatively flat. This demonstrates that token count and latency are related but not the same: on CPU-backed workloads, waiting often dominates end-to-end time.

Architectural point The front of the context is previously saved work (the cache/prefill). Each turn only pays for the delta it adds. That incremental shape — previously cached state at the front plus small deltas appended each turn — is the published trace for agent workloads in miniature. Prometheus metrics: measuring program-level cache hit concentration To quantify cache behavior across the fleet, we query Prometheus for prefix cache hits aggregated by pod. The results show concentrated program-level hits: across the 8-turn agent run, almost everything the servers were asked to read was already in cache, and one pod owned the bulk of the program’s cache. Example Prometheus output (prefix cache hits summed by pod over 5m):

sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))

{pod="optimized-baseline-cpu-vllm-decode-7b976466db-w26xt"} 7275.78947368421
{pod="optimized-baseline-cpu-vllm-decode-7b976466db-dwctz"} 0
{pod="optimized-baseline-cpu-vllm-decode-7b976466db-kvdsl"} 0

Load time: 3393ms    Result series: 3

Summary table (pod series and hits):

Pod	Prefix cache hits (5m increase)
`optimized-baseline-cpu-vllm-decode-7b976466db-w26xt`	7275.78947368421
`optimized-baseline-cpu-vllm-decode-7b976466db-dwctz`	0
`optimized-baseline-cpu-vllm-decode-7b976466db-kvdsl`	0

One pod owned the entire program-level cache for this run. This shows how program affinity can concentrate lookups and why routing and cache partitioning are critical design decisions. Also note: on CPU-based serving, overall latency is often dominated by compute and I/O, so token counts alone don’t predict wall-clock waiting.

Why this matters

The token counts and Prometheus queries together make the proof emphatic: program-level caching concentrates reads and reduces cross-pod duplication.
A single pod can end up owning the program for a multi-turn session, which is efficient for cache utilization but requires careful balancing to avoid hot spots.
These observations ground the architectural decisions you’ve seen earlier: splitting prefill from decode, using a router that respects program affinity, and designing fleet configs with observable metrics.

Where to go from here You started with an everyday question you couldn’t see inside, and you built the entire answer: models as files, GPUs/CPUs running them, servers sharing state, token loops producing program behavior, and caches preserving context across turns. Everything you need to reproduce these results — quick starts for building clusters, fleet shapes, and the benchmarks behind speed claims — is published at llm-d.ai. Further reading and references

Prometheus documentation: https://prometheus.io/docs/
Kubernetes basics: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
If you want to run inference in your own cluster, these resources and the examples above make serving language models at scale a practical, debuggable system rather than a mystery.

From ChatGPT to a Model Server

Inside a Single Request

From One Server to a Fleet

Meet llm-d

Tour the Running Stack

Smart Routing, Live

Getting More From the KV Cache

Serving Bigger Models With llm-d

Running It Day to Day

Seeing What Is Happening

Wrap Up: Agents and Beyond

Demo An Agent Style Multi Turn Workload

Watch Video