Multimodal Capabilities Text and Images Part 1

In this lesson we’ll explore multimodal capabilities of foundation models: how to send and receive more than just text (images, documents, video, and other media). Many modern applications require the model to ingest binary content or to produce media as part of the response—so understanding the expected input formats, request shapes, and response parsing is essential. Common multimodal use cases:

Ask the model “What’s in this image?” and get a concise description.
Supply a PDF and ask targeted questions about the content.
Provide a short video clip for summarization or to extract timestamps and captions.
Request image or video generation as part of a creative workflow.

A presentation slide titled "Problem: Modern GenAI Apps don't just take text" that shows four turquoise icons and examples of inputs/outputs. The examples read: "Text + Image for analysis," "PDF or document for question answering," "A video reference for summarization," and "Image or video output for creative generation."

Before integrating a model, verify what modalities it supports. Many models remain text-in/text-out only; others accept images, audio, or documents. Check the model catalog for supported input/output types and any required formats.

Always consult the model catalog to confirm input types (image, document, audio, video) and output types, plus any required metadata or formatting (for example: format, name, or source fields).

A presentation slide titled "Problem: Multimodal Apps need more than a Prompt String" showing a schematic "multimodal request anatomy" icon and a list of challenges (different input formats, limited model modality support, varied output types, and the need for correct packaging/parsing).

Consider how a model returns outputs:

Single-response models return the full result in one response.
Streaming-capable models return incremental chunks (text or media parts). Your application must handle chunked responses when using streaming endpoints or methods (for example, a ConverseStream or streaming variant of an invoke API).

Not all SDK methods support every workflow. Some models expose streaming APIs for incremental output while others only support non-streaming invoke-style calls. Plan routing, timeouts, and error handling based on the model’s capabilities.

Recommended approach

Use a small set of repeatable patterns to package inputs and parse outputs for each modality.
Route requests in your app by modality and by the selected model’s capabilities.
Use a conversational API (Converse) for multi-turn, contextful dialogs, or a one-shot InvokeModel-style call for single requests.
Keep request/response handling modular to enable swapping models without large refactors.

A dark-themed flowchart titled "Workflow: Building a Multimodal Request" showing linked steps from "Pick a multimodal-capable model" to "Build the request body," "Send binary content," then "Parse text/stream/media," "Present or store result," and "Add validation and fallback logic in the app."

Workflow summary (quick reference)

Step	Action
1	Pick a multimodal-capable model (consult the model catalog).
2	Build the request body and include any binary content (images, PDFs, video) in the supported format (`format`, `name`, `source.bytes`).
3	Send the request to the runtime (use the streaming endpoint if the model supports streaming).
4	Parse the response (single-response body, streaming text chunks, or binary media bytes).
5	Present or store the result; add validation and fallback logic to handle unexpected outputs.

Examples

Example 1 — Image in, Text out (local file) This example sends a local image file alongside a text prompt and uses a Converse call to ask the model to describe the image. The message content mixes text and a binary image object so you can swap models with minimal changes.

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

with open("invoice.png", "rb") as f:
    image_bytes = f.read()

response = client.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {"text": "What is this image showing? Summarize it in 3 bullet points."},
                {
                    "image": {
                        "format": "png",
                        "source": {"bytes": image_bytes}
                    }
                }
            ]
        }
    ]
)

# Safely extract text result (check response shape in case of model differences)
answer = response.get("output", {}).get("message", {}).get("content", [])
if answer:
    text_parts = [c.get("text") for c in answer if "text" in c]
    print("\n".join([p for p in text_parts if p]))
else:
    print("No text output returned; inspect the full response:", response)

Notes:

Ensure the image file is accessible to the runtime environment (local filesystem, mounted volume, or container).
Mixing text and an image object in a single message simplifies changing models or switching from local files to remote object stores.

Example 1 (variant) — Image in from S3, Text out Images often live in S3 (or another object store). Retrieve the image bytes from S3 and forward the bytes in the same message structure.

A stylized workflow diagram titled "Workflow: Example 1 Variant – Image From S3, Text Out" showing a cloud/browser icon in the center. It depicts an S3 bucket feeding a "Send" box (text question + image content) into a service and returning a "Receive" box with a text answer, with S3 and Bedrock SDK clients shown.

import boto3

# Clients
s3 = boto3.client("s3")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

# S3 location
bucket = "my-image-bucket"
key = "images/invoice.png"

# Read image from S3
s3_response = s3.get_object(Bucket=bucket, Key=key)
image_bytes = s3_response["Body"].read()

# Send to Bedrock Runtime via Converse
response = bedrock.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {"text": "What is this image showing? Summarize in 3 bullet points."},
                {
                    "image": {
                        "format": "png",
                        "source": {"bytes": image_bytes}
                    }
                }
            ]
        }
    ]
)

answer = response.get("output", {}).get("message", {}).get("content", [])
if answer:
    print(answer[0].get("text"))
else:
    print("No response text found; inspect full response:", response)

Example 2 — Document in (PDF), Text out Supplying a document (PDF) plus a question lets the model answer based on the document context. The example below sends a PDF’s bytes in the message content.

A stylized workflow diagram titled "Workflow: Example 2 – Document In, Text Out" showing two colored rounded panels labeled "Send" and "Receive" connected through a central speech-bubble icon. The left panel includes a small box labeled "Question / Document content."

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

with open("policy.pdf", "rb") as f:
    pdf_bytes = f.read()

response = client.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {"text": "Summarize the return policy in plain English."},
                {
                    "document": {
                        "format": "pdf",
                        "name": "policy",
                        "source": {"bytes": pdf_bytes}
                    }
                }
            ]
        }
    ]
)

answer = response.get("output", {}).get("message", {}).get("content", [])
if answer:
    # Models may return multiple content items; extract text parts.
    text_parts = [c.get("text") for c in answer if "text" in c]
    print("\n".join([p for p in text_parts if p]))
else:
    print("No text output returned; inspect the full response:", response)

Best practices and key takeaways

Confirm model capabilities in the model catalog (input/output modalities, required metadata).
Package binary content as the model expects (specify format, name, and provide source.bytes).
Implement robust parsing for both single-response and streaming outputs; handle partial or chunked data.
Add validation and fallback logic (e.g., if the model returns no text, fallback to retry or a simpler model).
Keep request/response handling modular to enable model or workflow changes without large refactors.

Additional resources

Amazon Bedrock Runtime Documentation
Amazon S3 Developer Guide
Model catalogs and SDK docs (check your platform/provider for the latest model capability listings and API details).

Introduction to Gen AI and LLMs

Introduction to Amazon Bedrock

Getting Started With Amazon Bedrock

Foundation Models in Amazon Bedrock

Basic Concepts of Prompt Engineering

Introduction to Text Generation With Amazon Bedrock

How Foundation Models Process Input

Extract Insights Using Prompt Engineering

Working With Different Task Types

Integrating Amazon Bedrock With Other AWS Services

Build a Marketing Email Generation Application

Introduction to Retrieval Augmentation With Bedrock Knowledge Bases

Best Practices and Optimization

Governance, Safety in Responsible AI

Security

Monitoring and Logging

Taking action with Bedrock Agents

Advanced Topics (Optional)

Final Project

Wrap Up

Multimodal Capabilities Text and Images Part 1

Examples

Watch Video

​Examples

Watch Video

Examples