Using Amazon CloudWatch With Bedrock Part 2

Metrics are useful for visualizing system state at a point in time. To be notified when those metrics change in meaningful ways, use CloudWatch Alarms. An alarm monitors a metric and triggers when the metric breaches a defined threshold or condition. For example, you can create an alarm that fires when the number of errors in a five-minute window exceeds 100. Alarms can be simple static thresholds or use more advanced evaluation methods (for example, anomaly detection). When an alarm triggers, you define the action: notify an on-call team, send a developer message, or start automation to remediate the issue without human intervention. Alarms continually evaluate the chosen metric — if the threshold is not breached they keep monitoring and re-evaluating.

CloudWatch Alarm example

In the CloudWatch Management Console, go to Metrics → Alarms → Create alarm. When creating an alarm you select the metric and any dimensions (for example, the model or namespace) that you want to watch.

A presentation slide titled "Workflow: Respond With CloudWatch Metric Alarms" showing the AWS CloudWatch "Create alarm" interface. The screenshot displays metric details (InputTokenCount for a model), a graph area, and condition settings with a threshold set to 8000.

In this example:

Namespace: AWS Bedrock
Metric: InputTokenCount for the model Meta Llama 3 8B Instruct
Statistic: 5-minute average
Condition: trigger when the metric is greater than 8000

You can attach targets to an alarm (SNS, Lambda, Systems Manager Automation, EventBridge, etc.), or leave it as a standalone monitor. In production, triggered alarms commonly start automated remediation workflows; in simpler setups they notify humans.

Alarms can be as simple or sophisticated as you need. Start with a conservative static threshold to reduce noise, then refine the alarm using anomaly detection or composite alarms as patterns emerge.

Quick steps to create a CloudWatch alarm

Open CloudWatch → Metrics → choose the Bedrock namespace.
Select the metric and any dimensions (model, region).
Click Create alarm → define the statistic (e.g., 5-minute average).
Set the threshold and evaluation period (e.g., > 8000 for 5 minutes).
Choose actions (SNS, Lambda, Systems Manager, or none).
Review and create the alarm.

Bedrock model invocation logging vs application logging

When building Bedrock-based applications, consider two logging layers:

Logging layer	What it captures	Typical use case
Model invocation logging	Full request and model response captured by Bedrock (platform-level)	Audit trails, replaying model interactions, debugging model outputs
Application logging	Logs you add in application code (request creation, business logic, user/context metadata, errors)	Correlation with user IDs, transaction IDs, processing checkpoints, higher-level observability

Both layers are important: Bedrock invocation logs record exactly what the Bedrock Runtime received and returned, while your application logs add context (user ID, transaction ID, processing stage) that helps operators troubleshoot and correlate events. Bedrock Model Invocation Logging is disabled by default. When you enable it in the Bedrock settings you choose:

Destination: Amazon S3, CloudWatch Logs, or both
Data types to capture: text, images, embeddings, video
An IAM service role that grants Bedrock permission to write to the destination

Be careful: enabling model invocation logging will capture request inputs and model outputs. If payloads contain secrets or personal data, ensure you comply with your security and privacy policies.

Enabling Bedrock model invocation logging records the full request and model response (input and output). Ensure you have appropriate data handling, retention, and access controls in place before enabling this feature.

In the Management Console, open Bedrock → Settings, toggle Model Invocation Logging on, pick the data types to capture, and select your destination. In this example we send logs to CloudWatch Logs and create a new log group named model-invocations. We also specify an IAM role that gives Bedrock permission to write into CloudWatch Logs.

A screenshot of an Amazon Bedrock settings page titled "Workflow: Bedrock Model Invocation Logging." It shows options to enable model invocation logging, pick data types (text, image, embedding, video), and choose logging destinations (S3, CloudWatch) and a service role.

Viewing Bedrock invocation logs in CloudWatch Logs

After enabling Bedrock invocation logging to CloudWatch Logs, go to CloudWatch → Logs → Log groups. You will see a log group (for example, model-invocations) that Bedrock writes to. Log events appear as timestamped messages. Example log lines:

2026-03-30T15:02:18.000Z {"timestamp":"2026-03-30T15:02:18Z","accountId":"485186561655","region":"us-east-1","requestId":"82a2c470-0463-4995-9151-0968a81ba4b8","operation":"ConverseStream", ...}
2026-03-30T15:04:17.000Z {"timestamp":"2026-03-30T15:04:17Z","accountId":"485186561655","region":"us-east-1","requestId":"6e91b2ee-a9a9-4a73-a12e-7251ee657f77","operation":"ConverseStream", ...}
2026-03-30T15:05:30.000Z {"timestamp":"2026-03-30T15:05:30Z","accountId":"485186561655","region":"us-east-1","requestId":"efc72073-0efb-4070-bd33-823ef09cfda3","operation":"ConverseStream", ...}
2026-03-30T15:26:26.000Z {"timestamp":"2026-03-30T15:26:26Z","accountId":"485186561655","region":"us-east-1","requestId":"ed3e3127-75e1-464d-939f-4b838acde1eb","operation":"ConverseStream", ...}
2026-03-30T15:27:41.000Z {"timestamp":"2026-03-30T15:27:41Z","accountId":"485186561655","region":"us-east-1","requestId":"76f5f3fb-4ff0-4b28-a558-b67bfbb37181","operation":"ConverseStream", ...}

Click into a log event to expand the full JSON payload for that invocation. Each event includes metadata (timestamp, account, region, requestId), the model identifier, the input payload, and the model response. Example payload:

{
  "timestamp": "2026-03-30T15:04:17Z",
  "accountId": "485186561655",
  "region": "us-east-1",
  "requestId": "6e91b2ee-a9a9-4a73-a12e-7251ee657f77",
  "operation": "ConverseStream",
  "modelId": "arn:aws:bedrock:us-east-1:485186561655:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0",
  "input": {
    "inputContentType": "application/json",
    "inputBodyJson": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "text": "what exactly is a token in llms"
            }
          ]
        },
        {
          "role": "assistant",
          "content": [
            {
              "text": "A **token** in Large Language Models (LLMs) is the fundamental unit of text that the model processes. Think of it as the \"atomic unit\" of language for AI systems.\n\n# What tokens look like:\n\nTokens aren't always whole words. They can be:\n\n*Complete words*: \"hello\" = 1 token\n*Parts of words*: \"running\" might be \"run\" + \"ning\" (2 tokens)\n*Punctuation*: \",\" = 1 token\n*Spaces and characters*: Even spaces count as tokens\n*Numbers*: \"2023\" might be 1 token or split into parts\n\n# Examples of tokenization:\n\n\"Hello world!\" -> [\"Hello\", \"world\", \"!\"] (3 tokens)\n\"AI is amazing\" -> [\"AI\", \"is\", \"amazing\"] (3 tokens)\n\"running\" -> [\"run\", \"ning\"] (2 tokens)\n\n# Why tokens matter:\n\n1. Processing: LLMs convert all text into tokens before understanding it\n2. Limitations: Most models have token limits (e.g., 4,000 or 8,000 tokens per conversation)\n3. Costs: Many AI services charge per token used\n4. Memory: Tokens determine how much context the model can remember\n\n# Key points:\n\n- **1 token ≈ 0.75 words** (rough estimate in English)\n- Common words are usually 1 token\n- Rare or complex words might be split into multiple tokens\n- Different languages have different tokenization patterns\n\nTokens are essentially how LLMs \"see\" and process language - they're the bridge between human text and machine understanding."
            }
          ]
        }
      ]
    }
  }
}

This demonstrates that Bedrock invocation logs capture the complete interaction (request and response) without requiring changes to your application code. Once you enable model invocation logging and grant the necessary IAM permissions, Bedrock will begin writing invocation logs to the destination you selected (CloudWatch Logs, S3, or both).

Best practices and operational tips

Minimize sensitive data in inputs if possible; redact or tokenise before sending.
Use application logs to add context (user IDs, transaction IDs) that are not present in model invocation logs.
Retention: set CloudWatch Logs retention policies to meet your compliance and cost objectives.
Automate: attach alarms to SNS or Lambda to shorten detection-to-remediation time.
Monitor token-based metrics (InputTokenCount, OutputTokenCount) to understand cost and usage patterns.

Introduction to Gen AI and LLMs

Introduction to Amazon Bedrock

Getting Started With Amazon Bedrock

Foundation Models in Amazon Bedrock

Basic Concepts of Prompt Engineering

Introduction to Text Generation With Amazon Bedrock

How Foundation Models Process Input

Extract Insights Using Prompt Engineering

Working With Different Task Types

Integrating Amazon Bedrock With Other AWS Services

Build a Marketing Email Generation Application

Introduction to Retrieval Augmentation With Bedrock Knowledge Bases

Best Practices and Optimization

Governance, Safety in Responsible AI

Security

Monitoring and Logging

Taking action with Bedrock Agents

Advanced Topics (Optional)

Final Project

Wrap Up

Using Amazon CloudWatch With Bedrock Part 2

CloudWatch Alarm example

Quick steps to create a CloudWatch alarm

Bedrock model invocation logging vs application logging

Viewing Bedrock invocation logs in CloudWatch Logs

Best practices and operational tips

Links and references

Watch Video

​CloudWatch Alarm example

​Quick steps to create a CloudWatch alarm

​Bedrock model invocation logging vs application logging

​Viewing Bedrock invocation logs in CloudWatch Logs

​Best practices and operational tips

​Links and references

Watch Video

CloudWatch Alarm example

Quick steps to create a CloudWatch alarm

Bedrock model invocation logging vs application logging

Viewing Bedrock invocation logs in CloudWatch Logs

Best practices and operational tips

Links and references