Implementing Conversational AI Part 1

• The core problem: foundation models are stateless
• The solution: structured conversations and message schemas
• Workflow: a basic chatbot loop
• Example: in-memory chat using Bedrock Runtime (boto3)
• Where to store conversation state (compare options)
• Session patterns and a simple session-tracking example
• Summary, best practices, and references

​

Problem: foundation models are stateless

• Recording what you send to the model (user messages, system prompts).
• Recording what the model returns (assistant replies, metadata).
• Re-sending the accumulated context on each new model call to enable multi-turn behavior.

​

Solution: structured conversations

1. Use a multi-turn/chat-style API (for example, the Converse method in the SDK).
2. Track an ordered conversation history (messages with role + content).
3. Optionally introduce orchestration (agents) or memory modules to extend capabilities.

​

Workflow: basic chatbot loop

1. User submits a message to your application.
2. Append the user message to the conversation history (with a role label).
3. Send the full conversation history to the model as the prompt:
   • For a brand-new session the history may contain only the system prompt and the first user message.
4. Model returns a response. Persist that assistant reply in the history.
5. Return the assistant reply to the user.
6. Repeat for each new user input: append, send full history, persist reply.

​

Example: simple chatbot with in-memory history

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

messages = []

def chat(user_input):
    # Append the user's message with a role label
    messages.append({
        "role": "user",
        "content": [{"text": user_input}]
    })

    # Call the model with the full conversation history
    response = client.converse(
        modelId="amazon.nova-lite-v1:0",
        messages=messages
    )

    # Extract the assistant's text reply from the response
    assistant_reply = response["output"]["message"]["content"][0]["text"]

    # Append the assistant reply to the local history
    messages.append({
        "role": "assistant",
        "content": [{"text": assistant_reply}]
    })

    return assistant_reply

# Example usage
print(chat("Hello"))
print(chat("What did I just say?"))

​

Where to store conversation state

• Session scoping: track a session_id for each conversation to map messages to sessions and support multiple concurrent sessions per user.
• Retention & pruning: conversation histories grow quickly — consider sliding windows, summarization, or TTL-based pruning.
• Security & privacy: avoid storing PII unnecessarily, encrypt data at rest and in transit, and enforce retention policies.

​

Sessions: why they matter

​

Simple pattern for session tracking (in-memory)

# session_messages maps session_id -> list of message dicts
session_messages = {}

def get_session_history(session_id):
    return session_messages.setdefault(session_id, [])

def chat_with_session(session_id, user_input):
    messages = get_session_history(session_id)

    messages.append({
        "role": "user",
        "content": [{"text": user_input}]
    })

    response = client.converse(
        modelId="amazon.nova-lite-v1:0",
        messages=messages
    )

    assistant_reply = response["output"]["message"]["content"][0]["text"]

    messages.append({
        "role": "assistant",
        "content": [{"text": assistant_reply}]
    })

    return assistant_reply

• Use a stable message schema across your app to make model swaps or multi-model orchestration straightforward.
• Consider a system prompt (role = system) to define assistant behavior consistently across sessions.
• Implement summarization or compression for long histories (e.g., summarize older sections into a compact form) to stay within token limits.
• Add explicit session timeouts and TTLs to limit storage costs and reduce exposure of sensitive data.
• Monitor model usage and cost; full-history resends increase token consumption.

​

Summary and next steps

• Foundation models are stateless: developers must track conversation history and re-send it to the model to achieve multi-turn behavior.
• Use a structured message schema (role + content) and append both user and assistant turns to a session-scoped history that you re-send on each model call.
• For production, persist histories using a datastore that matches your latency, durability, and scale requirements (DynamoDB, Redis, Aurora, etc.), adopt retention and privacy policies, and consider summarization strategies.
• Next in the series: agents/orchestration, memory summarization techniques, and scalable session management patterns.

• Amazon Bedrock documentation: https://docs.aws.amazon.com/bedrock/
• boto3 (AWS SDK for Python): https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
• DynamoDB: https://aws.amazon.com/dynamodb/
• Redis: https://redis.io/
• Amazon Aurora: https://aws.amazon.com/rds/aurora/