Demo The Token Crisis

In this lesson we simulate a common production problem for conversational AI: the token crisis. When you naively append every user and assistant turn to the prompt, the conversation history grows without bound. Over time this increases token counts, latency, and cost, and eventually exceeds the model’s context window — making the system slow, expensive, or unusable. This demo demonstrates how periodic memory summarization controls token growth. It uses plain Python (no external APIs), so you can focus on architecture and behavior rather than infrastructure. Overview

Problem: Unbounded prompt growth (the “token crisis”)
Solution: Periodic summarization to compress older context into long-term memory while keeping recent turns verbatim
Demo: Two simulations (naive vs managed) and a visual comparison

Step 1 — Setup Install the optional plotting package (matplotlib is used below):

# If needed in a fresh environment, uncomment:
!pip -q install matplotlib

Import required modules:

from typing import TypedDict, List, Dict
import math
import matplotlib.pyplot as plt

Step 2 — Shared state and token estimation We simulate a small conversational memory container similar to what agent frameworks use. The state contains:

messages: list of recent message dictionaries (role and content)
summary: compressed long-term memory (string)
turn_count: counter to decide when to summarize

For visualization we use a simple token estimator: roughly one token per four characters. This is an approximation for illustrative purposes only.

class ChatState(TypedDict):
    messages: List[Dict[str, str]]
    summary: str
    turn_count: int

def new_state() -> ChatState:
    return {
        'messages': [],
        'summary': '',
        'turn_count': 0,
    }

def estimate_tokens_from_text(text: str) -> int:
    # Rough approximation: ~1 token per 4 characters
    return math.ceil(len(text) / 4)

def estimate_state_tokens(state: ChatState) -> int:
    message_text = '\n'.join([f"{m['role']}: {m['content']}" for m in state['messages']])
    total_text = f"SUMMARY:\n{state['summary']}\n\nMESSAGES:\n{message_text}"
    return estimate_tokens_from_text(total_text)

Step 3 — Naive chatbot: append every turn forever This naive implementation appends both the user message and the assistant response to messages and increments turn_count. Nothing is removed or compressed.

def append_turn(state: ChatState, user_text: str, ai_text: str) -> ChatState:
    state['messages'].append({'role': 'human', 'content': user_text})
    state['messages'].append({'role': 'ai', 'content': ai_text})
    state['turn_count'] += 1
    return state

Step 4 — Simulate a long conversation (naive) We simulate a series of turns across several topics with intentionally verbose messages to make token growth visible.

topics = [
    'cloud architecture',
    'vector search',
    'retrieval augmented generation',
    'LLM evaluation',
    'cost optimization',
    'agent memory',
    'tool calling',
    'multi-agent routing',
    'guardrails',
    'prompt engineering',
    'streaming UX',
    'summarization strategies',
]

def make_user_message(i: int, topic: str) -> str:
    return (
        f"Turn {i}: Explain {topic} in detail, compare it with earlier concepts, "
        f"and keep the explanation practical for an enterprise chatbot that must scale."
    )

def make_ai_message(i: int, topic: str) -> str:
    return (
        f"On turn {i}, the assistant explains {topic}, relates it to previous topics, "
        f"adds trade-offs, risks, and examples, and provides a fairly long answer meant to simulate real token growth."
    )

Run the naive simulation and record estimated tokens after each turn:

naive_state = new_state()
naive_token_growth = []

for i in range(1, 13):
    topic = topics[(i - 1) % len(topics)]
    naive_state = append_turn(
        naive_state,
        make_user_message(i, topic),
        make_ai_message(i, topic),
    )
    naive_token_growth.append(estimate_state_tokens(naive_state))

print(naive_token_growth)  # estimated token counts after each turn

You should observe a steady upward curve: every turn increases the prompt size and estimated token count. This is the token crisis. Step 5 — Introduce a summarization node A practical solution is to periodically summarize older parts of the conversation into a compact summary, keeping only the most recent turns in full. The pattern is:

Compress older history into a summary string
Keep recent messages verbatim (short-term context)
Reset turn_count
Trigger summarization when turn_count reaches a threshold

Below is a deterministic demo summarizer that extracts topic-like fragments and builds a compact summary. In production, you would typically call a language model for higher-quality, semantic summarization.

def should_summarize(state: ChatState, threshold: int = 4) -> bool:
    # Summarize when the number of turns since the last summary reaches the threshold
    return state['turn_count'] >= threshold

def summarize_messages(state: ChatState) -> ChatState:
    if not state['messages']:
        return state

    recent_topics = []
    # Look back across recent messages for phrasing like "Explain <topic> in detail"
    for msg in state['messages'][-8:]:
        text = msg['content']
        part = None
        if 'Explain' in text and 'in detail' in text:
            # Extract the substring between 'Explain' and 'in detail'
            part = text.split('Explain', 1)[1].split('in detail', 1)[0].strip(' :')
        elif 'Explain' in text:
            # Extract the substring after 'Explain' if 'in detail' is not present
            part = text.split('Explain', 1)[1].strip(' :')
        else:
            # Fallback: take the first few words as a short descriptor
            part = ' '.join(text.split()[:6])

        recent_topics.append(part)

    # Remove duplicates while preserving order and form a compact summary
    compressed = ' . '.join(dict.fromkeys(recent_topics)) or 'previous discussion topics'
    state['summary'] = f'Conversation so far covered: {compressed}.'
    # Keep only the last 2 messages as the immediate short-term context
    state['messages'] = state['messages'][-2:]
    state['turn_count'] = 0
    return state

Periodic summarization is a common pattern in production chat architectures: compress older context into a summary while keeping recent turns verbatim to preserve immediate context.

This demo’s token estimator and summarizer are simplified for illustration. In production, use a model-based summarizer and an accurate tokenizer for your target LLM (e.g., tiktoken for OpenAI models).

Step 6 — Run the memory-aware (managed) version Run a second simulation that summarizes every 4 turns (threshold is tunable). We record token estimates and the summaries created.

smart_state = new_state()
smart_token_growth = []
summaries_created = []

for i in range(1, 13):
    topic = topics[(i - 1) % len(topics)]
    smart_state = append_turn(
        smart_state,
        make_user_message(i, topic),
        make_ai_message(i, topic),
    )

    if should_summarize(smart_state, threshold=4):
        smart_state = summarize_messages(smart_state)
        summaries_created.append((i, smart_state['summary']))

    smart_token_growth.append(estimate_state_tokens(smart_state))

print("Managed token growth:", smart_token_growth)
print("\nSummaries created:")
for turn, summary in summaries_created:
    print(f"After turn {turn}: {summary}")

With summarization, token growth becomes controlled: older parts of the conversation are compressed while recent turns remain detailed. This prevents the unbounded upward curve seen in the naive approach. Step 7 — Visual comparison (plot) Plot naive vs managed token growth to see the difference.

plt.figure(figsize=(9, 5))
plt.plot(range(1, len(naive_token_growth) + 1), naive_token_growth, marker='o', label='Naive: keep everything')
plt.plot(range(1, len(smart_token_growth) + 1), smart_token_growth, marker='s', label='Managed: summarize every 4 turns')
plt.xlabel('Turn')
plt.ylabel('Estimated Tokens in State')
plt.title('The Token Crisis vs Context Management')
plt.grid(True)
plt.legend()
plt.show()

Step 8 — Constructing the prompt: inject the summary + recent messages A production-ready prompt pattern combines:

A system instruction
A compressed summary (older context)
The most recent messages (short-term context) verbatim

This pattern preserves long-term context efficiently while keeping the active context small.

def build_prompt(state: ChatState) -> str:
    recent_messages = '\n'.join([f"{m['role']}: {m['content']}" for m in state['messages']])
    return (
        "SYSTEM: You are a helpful assistant.\n"
        f"SYSTEM: Conversation summary so far: {state['summary']}\n\n"
        f"RECENT TURNS:\n{recent_messages}\n\n"
    )

print(build_prompt(smart_state))

Example output (approximate):

SYSTEM: You are a helpful assistant.
SYSTEM: Conversation summary so far: Conversation so far covered: guardrails . prompt engineering . streaming UX . summarization strategies.

RECENT TURNS:
human: Turn 12: Explain summarization strategies in detail, compare it with earlier concepts, and keep the explanation practical for an enterprise chatbot that must scale.
ai: On turn 12, the assistant explains summarization strategies, relates it to previous topics, adds trade-offs, risks, and examples, and provides a fairly long answer meant to simulate this situation.

Step 9 — Takeaways

The token crisis: without context management, conversation history grows until it exhausts the model’s context window, increasing cost and latency.
Summarization-based memory: periodically compress older interactions into a compact summary and keep recent turns verbatim — this reduces tokens while preserving relevant context.
Production pattern: build prompts combining a system instruction, a compressed long-term summary, and a short-term verbatim buffer. This scales well across many turns.

Comparison: naive vs managed

Approach	Behavior	Pros	Cons
Naive (keep everything)	Append every turn verbatim	Simple to implement; full context available	Unbounded token growth, higher latency/cost, hits context window
Managed (periodic summarization)	Compress older turns into `summary`, keep recent turns verbatim	Controls token growth; reduces cost and latency; scales to many turns	Requires summarization logic and careful tuning of thresholds

Links and References

Kubernetes Basics (example reference)
Use tokenizer libraries such as tiktoken for accurate token counts with OpenAI models
Consider model-based summarizers or fine-tuned summarization prompts for production-quality memory

Memory design is one of the most important architectural choices for production conversational systems: it directly impacts cost, latency, and session lifetime. Use periodic summarization (or other memory management strategies) to avoid the token crisis and keep conversations scalable.

Course Introduction

Foundational Graph Concepts and Toolkit Setup

The Core Workflow: Nodes, Edges, and Routing

State Management and Iterative Loops

Context Management and Short-Term Memory

Long-Term Memory and Stateful Persistence

Human-in-the-Loop (HILT) Architectures

Advanced Control and Debugging UX

Conclusion

Demo The Token Crisis

Watch Video