> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Implementing Conversational AI Part 2
> Best practices for session storage, context window management, and scaling conversational AI using DynamoDB, summarization, and sliding window strategies
A session represents a single conversation between a user and your application. To maintain conversational state across turns you need a session store: a place to persist the ordered history of messages for each session. Common backend choices include relational databases (for example, [Amazon Aurora](https://aws.amazon.com/rds/aurora/)), key-value caches (for example, [Redis](https://redis.io/)), or document stores (for example, [Amazon DynamoDB](https://aws.amazon.com/dynamodb/)). The exact technology depends on your scale, latency, and operational requirements.
Typically, each conversation is assigned a unique session identifier (for example, a UUID or strings like `user-1`, `user-456`, `user-c`). That session ID is the lookup key in the session store, and the stored value is the ordered list of messages (roles: user or assistant, message content, timestamps, metadata).
Example: a minimal in-memory session store (illustrative only — not for production):
```python theme={null}
# sessions is an in-memory store mapping session_id -> list_of_messages
sessions = {}
def chat(session_id, user_input):
if session_id not in sessions:
sessions[session_id] = []
messages = sessions[session_id]
# Append the incoming user message to the session history
messages.append({
"role": "user",
"content": [{"text": user_input}]
})
# Call the conversational model with the accumulated history
response = client.converse(
modelId="amazon.nova-lite-v1:0",
messages=messages
)
# Extract the assistant reply text from the model response
reply = response["output"]["message"]["content"][0]["text"]
# Append the assistant reply to the session history
messages.append({
"role": "assistant",
"content": [{"text": reply}]
})
return reply
```
This pattern stores each session as a list of message objects under the session ID key, enabling quick retrieval of conversational context for the session. For production, persist sessions to durable storage (DynamoDB, Redis, etc.) and include operational metadata such as timestamps, TTLs, and indexing keys.
Context window and token limits
When you pass the full conversation history to a stateless foundation model at each turn, the history grows and consumes tokens. Foundation models have a fixed context window: the total input tokens plus expected output tokens must fit within that window. A typical message flow includes a system prompt, prior user messages, prior assistant responses, the current user message, and the expected model response. Each turn adds tokens, and eventually the history may exceed the model’s context window.
As the conversation progresses, input tokens increase. For illustration, an early turn might consume \~50 tokens, by turn five \~250 tokens, by turn twenty \~1.2k, and by turn fifty \~6k. As history grows the available space for model-generated output shrinks. If the context window is hit, older messages get truncated, the model loses earlier context, responses can become inconsistent, and requests may fail.
Mitigating context-window limits
You cannot keep appending the full history indefinitely. Common strategies to manage token growth:
* Truncation (sliding window): keep only the most recent N messages/turns.
* Summarization: compress older conversation segments into compact summaries that preserve important context.
* Hybrid strategies: combine selective truncation, summaries, and per-message indexing in a datastore.
A simple sliding-window example:
```python theme={null}
MAX_MESSAGES = 3
# Keep only the last MAX_MESSAGES messages in the history
messages = messages[-MAX_MESSAGES:]
```
Summarization is often more effective than aggressive truncation: periodically compress long segments of conversation into a short summary (for example, a paragraph) and store that summary alongside recent turns. On subsequent turns, pass the summary plus recent messages to the model to preserve key context while freeing token budget for the current interaction.
Quick comparison (useful for design decisions):
| Strategy | Strengths | When to use |
| - | - | - |
| Sliding window | Simple and low-cost | Short-lived sessions; stateless or limited storage |
| Summarization | Keeps key context with fewer tokens | Long-running sessions where historical context matters |
| Per-message storage + indexing | Selective retrieval and efficient reads | High-scale systems requiring partial history access |
Using DynamoDB for session storage
[Amazon DynamoDB](https://aws.amazon.com/dynamodb/) is a serverless, low-latency, highly scalable key-value and document store that often fits session-storage needs. However, DynamoDB’s query semantics (partition key + optional sort key) and item-size limits influence your table design.
A straightforward approach is to store an entire conversation as a single item keyed by `sessionId`. On each request you fetch the item, append new messages, and write it back. This is easy to implement but may hit DynamoDB item size limits as history grows.
Production considerations
Plan for operational requirements from the start. Consider these best practices:
* Timestamp every message for ordering and auditability.
* Use TTL (time-to-live) to expire stale sessions automatically.
* Summarize older turns proactively to control token growth.
* Store messages as individual items (one message per row) instead of a single large blob to enable selective retrieval and smaller reads/writes.
DynamoDB item example (storing an entire conversation as a single item):
```json theme={null}
{
"sessionId": "user-123",
"messages": [
{
"role": "user",
"content": [
{ "text": "Hello" }
]
},
{
"role": "assistant",
"content": [
{ "text": "Hi, how can I help?" }
]
}
]
}
```
Design your DynamoDB table keys and indexes around your access patterns. If you need to query by attributes other than `sessionId`, add appropriate secondary indexes or adapt your schema to support efficient reads.
DynamoDB operational limits and recommendations
Keep in mind DynamoDB’s constraints and operational trade-offs:
* A single DynamoDB item has a maximum size of 400 KB. Storing an ever-growing session as one item can hit this limit.
* Per-message items (one row per message) enable partial retrieval and reduce the likelihood of large writes.
* Implement TTL and summarization to avoid unbounded growth and to keep latency predictable.
* If you need to store large message payloads (attachments, long transcripts), consider external blob storage (S3) and reference keys in DynamoDB.
Avoid unbounded growth in stored session history. Implement TTLs, summarization, or per-message partitioning to prevent exceeding DynamoDB item size limits and to keep retrieval/write latency predictable.
Links and references
* [Amazon DynamoDB](https://aws.amazon.com/dynamodb/)
* [Amazon Aurora](https://aws.amazon.com/rds/aurora/)
* [Redis](https://redis.io/)
Further reading
* Designing for context windows and token budgets when building conversational AI
* Approaches to incremental summarization for long-running conversations
* DynamoDB best practices for time-series and per-item growth patterns
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.