Demo A Multi Turn Chat Reuses the Cache

​

Concept recap

• Models are stateless between requests: a typical chat turn resends the entire conversation (opening context, every question, every answer, plus the new question).
• That transcript grows with each turn, increasing how many tokens the server must read and (without caching) reprocess.
• Prefix caching prevents that growth from making each subsequent turn more expensive: previously-seen prefixes are cached and do not need reprocessing, only the newest tail requires fresh work.

​

Conversation setup

• We start with an opening context deliberately longer than one cache block so the cache will be exercised.
• Each loop appends a new question to the transcript, builds a JSON payload safely using jq, posts to the local completions endpoint, records timing and prompt token usage, then appends the model’s answer back into the transcript for the next loop.
• Using jq to construct the JSON payload avoids common demo failures from unescaped quotes, newlines, or other special characters in model output.

#!/usr/bin/env bash
CONVO="Conversation opener: (long deliberate context that spans more than one cache block)"
for i in $(seq 1 5); do
  CONVO="${CONVO} Question: What is one more important thing to know about serving these models well? Answer:"
  PAYLOAD=$(jq -n --arg prompt "${CONVO}" \
    '{model: "Qwen/Qwen2.5-0.5B-Instruct", prompt: $prompt, max_tokens: 60}')
  SECS=$(curl -s -o /tmp/turn.json -w '%{time_total}' \
    http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d "${PAYLOAD}")
  TOKENS=$(jq -r '.usage.prompt_tokens' /tmp/turn.json)
  ANSWER=$(jq -r '.choices[0].text' /tmp/turn.json | tr '\n' ' ')
  echo "turn ${i}: prompt tokens ${TOKENS}, took ${SECS}s"
  CONVO="${CONVO} ${ANSWER}"
done

​

Example run: single-turn output

turn 1: prompt tokens 143, took 4.129995s

​

Full five-turn run (prompt token counts and timings)

turn 1: prompt tokens 143, took 4.129995s
turn 2: prompt tokens 223, took 3.801295s
turn 3: prompt tokens 303, took 3.764113s
turn 4: prompt tokens 383, took 3.588462s
turn 5: prompt tokens 463, took 3.512344s

​

Observations

• Prompt tokens increase each turn as the transcript grows (143 → 463 in this run).
• Despite the transcript more than tripling in token count, per-turn latency barely decreases across turns. That indicates the cache is absorbing previously seen content: the server doesn’t re-read the cached prefix token-by-token, only the newest tail needs fresh processing.
• This behavior yields stable latency for long-lived chats when sessions are routed consistently to pods that hold the cached prefixes.

​

Validate cache behavior with Prometheus

sum(rate(vllm:prefix_cache_hits_total[5m])) /
sum(rate(vllm:prefix_cache_queries_total[5m]))

0.5914191419141914

sum by(pod) (increase(vllm:prefix_cache_hits_total[5m]))

{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-4hmpb"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-drbrd"} 0
{pod="optimized-baseline-cpu-vllm-decode-5c77c88766-g7kb"} 943.1578947368421

• The per-pod view shows where sessions are being served from. In this experiment a single pod owned the session; its cache hits increased while other pods showed none.

​

Quick reference

​

Summary

• The test shows that as the conversation grows, most of the transcript becomes cached and subsequent turns do not pay again for the cached history.
• A router that consistently sends a session back to the same pod plus an effective prefix cache keeps conversation performance stable even as the transcript grows.
• When fleet memory is limited, precise session routing and tiered storage strategies help maintain cache effectiveness at scale.

​

Links and references

• vLLM (prefix cache metrics): https://github.com/vllm-project/vllm
• jq: https://stedolan.github.io/jq/
• Prometheus: https://prometheus.io/