Managing Costs and Optimizing Performance Part 3

Choosing the right model and constraining its outputs are two of the most effective levers for controlling cost and latency. Use smaller models for classification, summarization, and other high-volume, low-complexity requests; reserve larger models for tasks that require deep reasoning, multi-step workflows, or external tool calls (for example, when the model must call APIs, query databases, or orchestrate multiple steps).

Model selection: match size to task

Small models: fast, low-cost, ideal for classification, sentiment analysis, short summaries, and deterministic transformations.
Large models: higher latency and cost, best for complex reasoning, multi-step tasks, and situations requiring broad context or advanced synthesis.
When unsure, benchmark both representative smaller and larger models on your workload and measure accuracy, latency, and cost.

Control output size and format

Explicitly constraining output length and format reduces token usage and prevents verbose, costly responses. When invoking a model, set a hard token limit and provide a strict output template.

A presentation slide titled "Workflow: Control Output Size" showing three numbered tips: 01 cap with maxTokens, 02 use clear instructions, and 03 use constrained formats (e.g., JSON, bullets). Each tip is displayed in a colored boxed panel with brief explanations.

Example: limit tokens and ask for a concise format using the Bedrock runtime (boto3):

import boto3
import json

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

prompt = "Summarize the benefits of cloud computing in 3 bullet points."

response = bedrock.invoke_model(
    modelId="amazon.nova-lite-v1:0",
    body=json.dumps({
        "inputText": prompt,
        "textGenerationConfig": {
            "maxTokenCount": 100,    # limit output size
            "temperature": 0.5
        }
    }),
    contentType="application/json",
    accept="application/json"
)

result = json.loads(response["body"].read())
print(result["results"][0]["outputText"])

In the example above, maxTokenCount is set to 100. Coupled with an instruction that the output must be exactly three bullet points, this prevents rambling and reduces inference cost.

Prompt design: reduce input token cost

The input context also contributes to cost. Avoid repeatedly sending large or irrelevant documents. In retrieval-augmented generation (RAG), only pass the most relevant chunks into the prompt to keep both input and output token usage low.

A slide titled "Workflow: Prompt Design Impacts Cost" listing three costly mistakes: repeating large context unnecessarily, injecting irrelevant documents in RAG, and sending entire documents instead of chunks. Each item appears in an orange-outlined box with an X icon on a dark blue background.

Naive approach — embedding an entire document in the prompt (costly):

full_document = """
Returns are accepted within 30 days...
Shipping takes 5-7 days...
Warranty lasts 1 year...
[...very long document...]
"""

prompt = f"""
Use the following document to answer the question:

{full_document}

Question: What is the returns policy?
"""

Cost-effective approach — retrieve only relevant chunks and pass a concise context:

full_document = """
Returns are accepted within 30 days...
Shipping takes 5-7 days...
Warranty lasts 1 year...
[...very long document...]
"""

# Assume retrieval returns only the relevant passages:
retrieved_chunks = [
    "Items can be returned within 30 days of delivery.",
    "Refunds are issued to the original payment method."
]

context = "\n".join(retrieved_chunks)

prompt = f"""
Use the following context to answer the question:

{context}

Question: What is the returns policy?
"""

By retrieving and sending only the relevant paragraphs you reduce input token size and typically improve grounding and accuracy. For more on RAG concepts, see Retrieval-augmented generation and the Amazon Bedrock documentation.

When you require a strict structured output (for example JSON), include the exact schema or a template in the prompt. This reduces malformed outputs and prevents extraneous text that consumes tokens. For example, supply a JSON schema and instruct: “Return only valid JSON matching this schema.”

Knowledge base (RAG) billing and chunk-size trade-offs

When you use managed RAG services or Bedrock Knowledge Bases, costs arrive from multiple places. Monitor and optimize across the whole pipeline:

Billing dimension	What it covers	Optimization tips
Ingestion	Embedding calls, chunking, parsing	Choose embedding model and chunk size wisely; batch embeddings to reduce overhead
Retrieval	Vector store query and storage costs	Select the appropriate vector store for your scale and query patterns
Inference	Model compute for the augmented prompt	Minimize augmented prompt size and pick the right foundation model for the task

Chunk size directly affects both ingestion and runtime costs:

Larger chunks → fewer embeddings (lower ingestion cost) but coarser retrieval. Retrieved chunks may include irrelevant information, increasing augmented prompt size and runtime cost.
Smaller chunks → more embeddings (higher ingestion cost) but more precise retrieval. Precise retrieval injects only tightly relevant content into the prompt and can lower runtime cost.

A slide titled "Workflow: Manage Knowledge Base Costs" comparing larger chunks (fewer embeddings, coarse retrieval, higher runtime cost) with smaller chunks (more embeddings, precise retrieval, lower runtime cost).

Evaluate trade-offs carefully: choose chunk sizes and embedding models that balance ingestion vs. runtime costs for your workload. Continuously monitor costs across ingestion, retrieval, and inference and adjust parameters based on observed accuracy and spend.

Quick reference

Topic	Recommendation
Use smaller models when possible	For high-volume, low-complexity tasks (classification, short summaries)
Constrain outputs	Use `maxTokenCount`, templates, and strict format instructions
Prompt design	Retrieve and send only relevant context; avoid duplicating large documents
RAG pipeline	Monitor ingestion, retrieval, and inference costs and tune chunk sizes and embedding models

Summary: favor smaller models for simple tasks, reserve large models for complex reasoning, constrain outputs (token limits and strict formats), design precise prompts, and use RAG intelligently—retrieve only necessary chunks to reduce token usage and cost.

Introduction to Gen AI and LLMs

Introduction to Amazon Bedrock

Getting Started With Amazon Bedrock

Foundation Models in Amazon Bedrock

Basic Concepts of Prompt Engineering

Introduction to Text Generation With Amazon Bedrock

How Foundation Models Process Input

Extract Insights Using Prompt Engineering

Working With Different Task Types

Integrating Amazon Bedrock With Other AWS Services

Build a Marketing Email Generation Application

Introduction to Retrieval Augmentation With Bedrock Knowledge Bases

Best Practices and Optimization

Governance, Safety in Responsible AI

Security

Monitoring and Logging

Taking action with Bedrock Agents

Advanced Topics (Optional)

Final Project

Wrap Up

Managing Costs and Optimizing Performance Part 3

Model selection: match size to task

Control output size and format

Prompt design: reduce input token cost

Knowledge base (RAG) billing and chunk-size trade-offs

Quick reference

Watch Video

​Model selection: match size to task

​Control output size and format

​Prompt design: reduce input token cost

​Knowledge base (RAG) billing and chunk-size trade-offs

​Quick reference

Watch Video

Model selection: match size to task

Control output size and format

Prompt design: reduce input token cost

Knowledge base (RAG) billing and chunk-size trade-offs

Quick reference