Implementing a Message Summarization Node

Summarization is a critical technique for managing token limits in long-running conversations with large language models (LLMs). Rather than discarding prior messages, a summarization node compresses them into a compact, high-value representation that preserves essential context while trimming verbosity. This reduces token usage, lowers cost, and improves model reliability. A message summarization node in LangGraph is a modular function that ingests a list of messages and returns a condensed representation. You can inject that summary back into the graph state to replace or augment the full chat history, keeping downstream nodes efficient and focused.

Imagine Robbie, who maintains pages of delivery logs. Before passing his notes to the next person, he rewrites the important parts onto a single sticky note. That sticky note is the summary: compact, actionable, and easy to carry forward.

You don’t need to summarize after every turn. Trigger summarization strategically — for example, at the end of a task cycle, before calling an LLM with tight token constraints, or after multiple turns of accumulated history. This targeted approach preserves compute and avoids unnecessary overhead.

Why a Summarization Node?

When conversations expand, simple trimming or filtering can lose context. A summarization node compresses the conversation into a smaller, information-dense representation so the graph continues to operate with sufficient context without exceeding token budgets. Key benefits:

Keeps long-running workflows efficient.
Reduces token consumption and cost.
Preserves essential facts, objectives, and action items.
Enables historical context to be reloaded compactly (e.g., from a vector DB).

Typical Implementation Flow

Instantiate a chat model to generate summaries.
Define the graph state schema (e.g., messages + summary).
Implement a node that builds a prompt (system instruction + conversation) and asks the model to produce a concise summary.
Store the summary in state and optionally reset or trim messages.

Example implementation (Python). Adjust imports, types, and the model invocation to match your SDK or runtime.

from typing import List, TypedDict
from langchain.schema import HumanMessage, SystemMessage, AIMessage
from langchain.chat_models import ChatOpenAI

# Setup model (adjust model_name/parameters for your SDK)
model = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.0)

# Define graph state schema
class GraphState(TypedDict):
    messages: List[HumanMessage | AIMessage]
    summary: str

# Summarization node
def summarize_messages(state: GraphState) -> GraphState:
    """
    Build a system prompt asking the model to summarize the conversation,
    then call the chat model with the combined messages.
    """
    prompt = [SystemMessage(content="Summarize the following conversation:")] + state["messages"]

    # Many chat clients return a single AIMessage or a list; adapt the call to your SDK.
    # Here we use predict_messages which returns a list of messages; take the first AIMessage.
    response = model.predict_messages(prompt)
    ai_message = None
    for msg in response:
        if isinstance(msg, AIMessage):
            ai_message = msg
            break
    summary_text = ai_message.content if ai_message else ""

    return {
        "summary": summary_text,
        "messages": [],  # Optionally clear or trim messages after summarization
    }

The prompt begins with a clear system instruction such as “Summarize the following conversation:” followed by the message history. The model returns a summary you store in the graph state and then optionally reset or trim the messages list. This compresses the conversation so it stops growing unbounded while preserving context.

When to Trigger Summarization

Use summarization intentionally at transition points that meaningfully reduce context size or before expensive downstream steps.

Trigger Point	Why it helps
End of a task cycle	Consolidates final decisions and outcomes for the next cycle
Before calling a memory- or compute-heavy LLM	Keeps the prompt within token limits
After N conversational turns	Prevents accumulation of low-value chit-chat
Prior to persisting to long-term storage	Stores compressed, relevant history

State Management Patterns

After creating a summary, you can choose how to store or apply it:

Storage Strategy	When to use	Example
Overwrite messages with summary	When raw history is unneeded or costly	`{"summary": "...", "messages": []}`
Keep both summary and raw messages	For auditing or debugging	`{"summary": "...", "messages": [...]}`
Maintain summary + truncated messages	Preserve recent turns + compressed history	`{"summary": "...", "messages": recent_messages}`

Advanced Summarization Strategies

To improve fidelity and downstream utility, consider:

Incremental summaries: update an existing summary after each turn rather than re-summarizing the whole history.
Role-based summaries: generate separate summaries for user, assistant, and tool messages to preserve perspective.
Structured summaries: produce JSON or key-value summaries to simplify downstream parsing and filtering.
Combine summarization with trimming: summarize the portion you plan to remove, then trim it from the message list.
Persist summaries in a vector store or other long-term storage to reload compact history for new sessions.

For working with persistent storage and vector databases, see resources on vector DBs and persistent storage strategies, such as vector databases for GenAI.

Monitoring and Improving Summaries

Summarization is not infallible. Implement monitoring and feedback to detect and correct information loss:

Log summary outputs and compare them periodically to raw history.
Add explicit prompt instructions to preserve critical facts, goals, and action items (e.g., “Preserve any user goals and action items”).
Use structured formats (JSON) to make validation and downstream processing deterministic.
Consider human-in-the-loop review for mission-critical flows.

Over time, iterate on prompts and summarization cadence so summaries become both smaller and more informative — saving tokens while preserving critical context.

Trigger summarization at meaningful transition points (end of task cycles, before expensive reasoning, or when history size exceeds thresholds) to balance compute cost and context fidelity.

Always validate summaries periodically. Important facts can be lost in aggressive compression — monitor, log, and tune prompts accordingly.

Course Introduction

Foundational Graph Concepts and Toolkit Setup

The Core Workflow: Nodes, Edges, and Routing

State Management and Iterative Loops

Context Management and Short-Term Memory

Long-Term Memory and Stateful Persistence

Human-in-the-Loop (HILT) Architectures

Advanced Control and Debugging UX

Conclusion

Implementing a Message Summarization Node

Why a Summarization Node?

Typical Implementation Flow

When to Trigger Summarization

State Management Patterns

Advanced Summarization Strategies

Monitoring and Improving Summaries

Watch Video

​Why a Summarization Node?

​Typical Implementation Flow

​When to Trigger Summarization

​State Management Patterns

​Advanced Summarization Strategies

​Monitoring and Improving Summaries

Watch Video

Why a Summarization Node?

Typical Implementation Flow

When to Trigger Summarization

State Management Patterns

Advanced Summarization Strategies

Monitoring and Improving Summaries