> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo A Multi Turn Chat Reuses the Cache

> Demonstrates prefix caching in multi-turn chats so growing conversation history doesn't increase per-turn processing cost, using a five-turn demo, jq payloads, and Prometheus metrics.

Everything in this module centers on one promise:

A conversation that comes back shouldn't pay for its own history.

This lesson demonstrates that promise with a five-turn chat. You'll see how prompt token counts grow while per-turn latency stays stable, thanks to prefix caching.

This demo illustrates prefix caching: previously-seen prompt prefixes are cached so the model doesn't reprocess them token-by-token on every turn. That keeps repeated conversation turns inexpensive even as the transcript grows.

## Concept recap

* Models are stateless between requests: a typical chat turn resends the entire conversation (opening context, every question, every answer, plus the new question).
* That transcript grows with each turn, increasing how many tokens the server must read and (without caching) reprocess.
* Prefix caching prevents that growth from making each subsequent turn more expensive: previously-seen prefixes are cached and do not need reprocessing, only the newest tail requires fresh work.

## Conversation setup

* We start with an opening context deliberately longer than one cache block so the cache will be exercised.
* Each loop appends a new question to the transcript, builds a JSON payload safely using [jq](https://stedolan.github.io/jq/), posts to the local completions endpoint, records timing and prompt token usage, then appends the model's answer back into the transcript for the next loop.
* Using `jq` to construct the JSON payload avoids common demo failures from unescaped quotes, newlines, or other special characters in model output.

Here is the full shell loop used to run five turns. Note how `jq` is used to build a safe JSON payload from the transcript rather than attempting to hand-glue text into JSON.

```bash theme={null}
#!/usr/bin/env bash
CONVO="Conversation opener: (long deliberate context that spans more than one cache block)"
for i in $(seq 1 5); do
  CONVO="${CONVO} Question: What is one more important thing to know about serving these models well? Answer:"
  PAYLOAD=$(jq -n --arg prompt "${CONVO}" \
    '{model: "Qwen/Qwen2.5-0.5B-Instruct", prompt: $prompt, max_tokens: 60}')
  SECS=$(curl -s -o /tmp/turn.json -w '%{time_total}' \
    http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "${PAYLOAD}")
  TOKENS=$(jq -r '.usage.prompt_tokens' /tmp/turn.json)
  ANSWER=$(jq -r '.choices[0].text' /tmp/turn.json | tr '\n' ' ')
  echo "turn ${i}: prompt tokens ${TOKENS}, took ${SECS}s"
  CONVO="${CONVO} ${ANSWER}"
done
```

Note: Using `jq` to construct the request ensures proper escaping of any quotes, newlines, or other characters in the model's answers. This avoids common demo failures caused by hand-gluing untrusted text into JSON.

## Example run: single-turn output

```text theme={null}
turn 1: prompt tokens 143, took 4.129995s
```

## Full five-turn run (prompt token counts and timings)

```text theme={null}
turn 1: prompt tokens 143, took 4.129995s
turn 2: prompt tokens 223, took 3.801295s
turn 3: prompt tokens 303, took 3.764113s
turn 4: prompt tokens 383, took 3.588462s
turn 5: prompt tokens 463, took 3.512344s
```

## Observations

* Prompt tokens increase each turn as the transcript grows (143 → 463 in this run).
* Despite the transcript more than tripling in token count, per-turn latency barely decreases across turns. That indicates the cache is absorbing previously seen content: the server doesn't re-read the cached prefix token-by-token, only the newest tail needs fresh processing.
* This behavior yields stable latency for long-lived chats when sessions are routed consistently to pods that hold the cached prefixes.

## Validate cache behavior with Prometheus

To confirm cache behavior across the fleet, check the prefix cache hit ratio (hits / queries). As the conversation grows, this ratio should climb toward 1 — meaning most prefix tokens were served from cache.

Prometheus query (ratio):

```promql theme={null}
sum(rate(vllm:prefix_cache_hits_total[5m])) /
sum(rate(vllm:prefix_cache_queries_total[5m]))
```

These metrics are exported by vLLM ([https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)).

Example output:

```text theme={null}
0.5914191419141914
```

A rising ratio indicates an increasing share of each prompt was served from cache rather than recomputed.

Prometheus query (per-pod increase in cache hits over 5m):

```promql theme={null}
sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))
```

Example per-pod output:

```text theme={null}
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-g7kb"} 943.1578947368421
```

* The per-pod view shows where sessions are being served from. In this experiment a single pod owned the session; its cache hits increased while other pods showed none.

## Quick reference

| Item | Purpose | Example / Notes |
| - | - | - |
| Metric to check overall cache effectiveness | Measure prefix cache hit ratio cluster-wide | Use the query: `sum(rate(vllm:prefix_cache_hits_total[5m])) / sum(rate(vllm:prefix_cache_queries_total[5m]))` |
| Metric to check which pod served the session | See per-pod increase in cache hits over a window | Use the query: `sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))` |
| Safe JSON payload construction | Avoid JSON-breaking characters in model output | Use `jq` to build payloads (see script above) |

## Summary

* The test shows that as the conversation grows, most of the transcript becomes cached and subsequent turns do not pay again for the cached history.
* A router that consistently sends a session back to the same pod plus an effective prefix cache keeps conversation performance stable even as the transcript grows.
* When fleet memory is limited, precise session routing and tiered storage strategies help maintain cache effectiveness at scale.

Using a safe JSON builder like [jq](https://stedolan.github.io/jq/) to construct request payloads is important in demos — model-generated text can contain quotes, newlines, and other characters that break hand-built JSON.

## Links and references

* vLLM (prefix cache metrics): [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
* jq: [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)
* Prometheus: [https://prometheus.io/](https://prometheus.io/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/llm-d-a-deep-dive/module/d1f497ec-31bd-44d3-ada0-2e441d7eceb8/lesson/8dda7366-2454-40bd-8767-b1279d7dca3b)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.