> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CloudWatch Logs and Log Insights Beyond the Basics

> Deep dive into Amazon CloudWatch Logs and Logs Insights covering architecture, ingestion methods, retention, security, metric and subscription filters, querying, cost optimization, fan‑out patterns, and cross-account integration.

In this lesson we go deeper into CloudWatch Logs and CloudWatch Logs Insights. Topics covered:

* CloudWatch Logs architecture and data flow.
* Common ingestion methods and trade-offs.
* Retention, storage, and cost-optimization strategies.
* Metric filters, subscription filters, and fan‑out patterns.
* Querying at scale with Logs Insights.
* Integration patterns, cross-account aggregation, and best practices.

Note: example pricing figures are directional and change frequently. Treat numbers below as illustrative only.

![A slide titled "CloudWatch Logs Architecture" showing a three-layer colored pyramid that illustrates the hierarchy: Log Groups (top), Log Streams (middle), and Log Events (bottom). The caption reads that it structures log data using a simple three-layer hierarchy.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/cloudwatch-logs-architecture-pyramid-hierarchy.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=b1578786447f8d2d2eee7932f9c420ca)

CloudWatch Logs uses a simple three-layer hierarchy:

* Log Groups — top-level logical grouping (for example, a service, application, or environment).
* Log Streams — ordered sequences of log events within a group (commonly time-boxed or per-instance streams).
* Log Events — the individual log entries.

Example: a payment-processor Lambda might have one log group per function, log streams per invocation window or instance, and many log events inside each stream.

Ingestion methods and trade-offs

Choose an ingestion approach that fits your infrastructure, latency, and operational constraints. The table below summarizes common options and trade-offs.

| Ingestion Method | Best For | Pros | Cons |
| - | - | - | - |
| Unified CloudWatch Agent | EC2 / on-prem / host-based systems | Handles multi-line logs, buffering, backoff, and retries | Agent management required; not ideal for ephemeral containers without a host agent |
| Direct API calls (PutLogEvents) | Microservices, containers that emit logs directly | Low latency, fine-grained control | Must handle batching, size/throughput limits, and sequence tokens; use SDKs to simplify |
| Service-native integrations | AWS-managed services (Lambda, ELB, VPC Flow Logs) | Easiest to enable; no extra agent code | Limited to services that support native integration |
| Container logging (fluentd, Fluent Bit) | Kubernetes, ECS | Centralized, can transform and route logs before ingest | Requires additional components; must forward to CloudWatch or another sink |

Key ingestion details:

* Direct API ingestion requires managing PutLogEvents constraints: batch size limits, throughput quotas, and sequence tokens. Use official AWS SDKs or community libraries that handle retries and token management.
* All ingestion methods require appropriate IAM permissions. Apply least-privilege policies and use resource-based policies for cross-account writes when needed.

Retention policies, storage, and security

* Retention is configured at the log group level; you cannot set retention per log stream or per event. Every stream and event in a log group inherits the group's retention setting.
* Choose retention by balancing compliance, operational usefulness, and cost. Options include days, months, years, or never expire.
* Encrypt logs at rest using AWS KMS (AWS-managed or customer-managed keys) and keep TLS in transit.

Retention is set at the log group level. All streams and events within a log group inherit that retention policy.

| Typical pricing components (directional examples) | Notes |
| - | - |
| Ingestion | \~`$0.50` per GB ingested (example only) |
| Storage | \~`$0.03` per GB‑month for retained logs (example only) |
| Logs Insights | Charged per GB scanned by queries (example `~$0.005` per GB scanned) |
| Encryption at rest | KMS usage may add costs if using customer-managed keys |
| Transit | Sent over TLS by default; VPC endpoints can reduce egress exposure |

Security best practices:

* Encrypt sensitive logs at rest with KMS and use TLS for transit.
* Use least-privilege IAM roles and resource-based policies for cross-account sharing.
* Use VPC endpoints (PrivateLink) for private connectivity to CloudWatch Logs when required.
* Enable CloudTrail to audit configuration changes and who accessed logs.

![A presentation slide titled "Encryption and Security" showing five best-practice tips. The tips recommend KMS encryption for sensitive logs, least‑privilege IAM, resource‑based cross‑account policies, enabling CloudTrail auditing, and using IAM conditions to restrict access.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/encryption-security-kms-iam-cloudtrail-practices.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=75515f708bbd1ede26d49b011d45b2f3)

Metric filters — creating metrics from logs

Metric filters let you convert log events into CloudWatch metrics in near real time without writing custom code. Use them to count occurrences, extract numeric values, and power alarms or dashboards.

Important characteristics:

* Metrics are generated only from new events as they arrive. Metric filters do not retroactively process historical logs unless you re-ingest them.
* There is no extra processing charge for metric-filter conversions beyond the normal ingestion and storage costs; however, the CloudWatch metrics produced are standard CloudWatch metrics and may incur metric-related charges.
* Limit: 100 metric filters per log group.
* Attach metric filters to appropriately organized log groups so filters produce meaningful metrics.

![A slide titled "Metric Filters – Creating Metrics From Logs" showing a funnel icon and an arrow that illustrates transforming unstructured log data into actionable CloudWatch metrics.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/metric-filters-logs-to-cloudwatch-metrics.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=8e830b071c31f76d2c71f4ba265a4b72)

![A presentation slide titled "Metric Filters – Creating Metrics From Logs" showing a four-part circular infographic of key capabilities: Pattern Matching, Namespaces, Dimensions, and Stats. Each quadrant is color-coded and labeled with brief descriptions of how logs are converted into metrics.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/metric-filters-pattern-namespaces-dimensions-stats.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=71f701bf115a39650cec567b6dbbf5c3)

![A presentation slide titled "Metric Filters – Creating Metrics From Logs" showing two info cards: one says "\$0 Processing Cost" and the other "100 Filters per Group" with brief explanatory notes beneath each.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/metric-filters-creating-metrics-from-logs.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=bcd51a1381a21568770c996787022bf7)

Subscription filters and fan-out patterns

A critical production and exam constraint: each log group supports only one subscription filter. Subscription filters stream log events in near-real time to one destination:

* Kinesis Data Streams
* Lambda
* Kinesis Data Firehose

Because of the one-filter-per-log-group limit, common multi-destination patterns include:

* Kinesis Data Firehose as the single subscription consumer, with Firehose delivering to multiple sinks (S3, OpenSearch, external HTTP endpoints) or invoking processing steps.
* Kinesis Data Streams for real-time fan-out where multiple consumers can independently read from the stream.
* A Lambda consumer that fans out to multiple downstream systems (S3, OpenSearch, third‑party APIs, etc.).

Subscription filters support cross-account delivery using appropriate IAM roles and resource policies.

Each log group supports only one subscription filter. To deliver to multiple destinations, use a fan-out pattern (Kinesis Streams, Firehose, or Lambda forwarding).

![A presentation slide titled "Subscription Filters – The ONE Filter Limitation" with a four-step timeline listing: 01 Destinations, 02 One Filter Rule, 03 Fan-Out Pattern, and 04 Cross-Account Delivery. Each step briefly notes that logs can stream to Kinesis Streams/Lambda/Firehose, that each log group supports only one subscription filter, using Kinesis for fan-out, and support for cross-account delivery.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/subscription-filters-one-filter-limit.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=4b418c80af922b814b42c45b9f8106b7)

Distinguishing Kinesis Streams vs Firehose:

* Kinesis Data Streams: best for low-latency, real-time processing and fan-out to multiple consumers.
* Kinesis Data Firehose: best for durable, batched delivery to S3, OpenSearch, or third‑party endpoints and for built-in buffering and transformation.

CloudWatch Logs Insights — querying at scale

CloudWatch Logs Insights offers a powerful query language to search, filter, extract fields, aggregate, and visualize results across log groups. Pricing is typically based on data scanned by a query, so minimizing scanned volume reduces cost.

Example Logs Insights query:

```sql theme={null}
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)
| sort @timestamp desc
```

Tips to reduce Logs Insights costs:

* Limit the date range and scope to the smallest set of log groups necessary.
* Filter early in the query pipeline to reduce scanned data.
* Use structured logging (JSON fields) so you can extract precise fields without scanning entire messages.
* Consider exporting older logs to S3 and running occasional ad-hoc analytics there instead of frequent large-scale Logs Insights scans.

![A presentation slide titled "CloudWatch Logs Insights – Querying at Scale." It shows four colorful icons and labels illustrating features: Searching logs, Filtering events, Extracting fields, and Aggregating and visualizing results.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/cloudwatch-logs-insights-querying-scale.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=cc927a2fe73d684d4b6bb7fd3585c2f4)

Cost optimization and data lifecycle

Practical strategies to manage cost while preserving observability:

* Selective retention — keep short retention for noisy, low-value logs and longer retention for critical logs.
* Export and archive — offload cold logs to S3 and use lifecycle rules to Glacier/Deep Archive for long-term storage.
* Log sampling — ingest a percentage (for example `1–10%`) of high-volume, low-value events instead of everything.
* Structured logging — emit JSON to enable precise field extraction and efficient queries.
* Early filtering — drop or redact low-value or sensitive events before ingestion or before querying.

![A slide titled "Cost Optimization Strategies" showing four colorful rounded panels labeled Selective Retention, S3 Export for Archives, Log Sampling, and Structured Logging, each with a short tip and an icon. The panels are blue, orange, green, and purple respectively.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/cost-optimization-selective-retention-s3-sampling.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=a63a21f34e5b742de24c559894c0c7a4)

Integration patterns and cross-account aggregation

Common production needs:

* Centralize logs from multiple AWS accounts into a security or central logging account.
* Consolidate logs across Regions.
* Forward logs to SIEMs or third-party logging platforms (Splunk, Datadog, Sumo Logic).
* Store and index logs in OpenSearch for advanced search and fuzzy matching beyond Logs Insights.

Typical cross-account aggregation pattern:

* Source account log groups stream to a subscription consumer (for example, a Firehose or Lambda) that assumes an IAM role with permission to write to the destination account’s S3 or OpenSearch.
* Ensure resource-based policies and KMS key policies (if encrypting) allow the intended cross-account principals.

![An infographic titled "Integration Patterns" showing a colorful semicircular diagram around a cloud icon that lists four log-integration options: Cross-Account Aggregation, Multi-Region Consolidation, SIEM Integration, and OpenSearch Integration, each with an icon and brief description.](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/integration-patterns-cloud-log-options.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=c71822daf95627da8866e4594418a99f)

Best practices and exam tips

* Use structured logging (JSON) and consistent naming/semantics across services.
* Apply appropriate log levels (DEBUG/INFO/WARN/ERROR/CRITICAL); avoid logging PII—mask or redact if necessary.
* Use selective retention and S3 lifecycles instead of relying on CloudWatch for indefinite archiving.
* Account for the one subscription filter per log group limit — design fan-out using Kinesis Streams, Firehose, or Lambda forwarding.
* Create billing alarms to monitor ingestion and storage costs.
* When using KMS encryption, ensure key policies allow the required principals to avoid accidental access issues (common exam trap).

![A presentation slide titled "Best Practices and Exam Tips" showing three colorful panels labeled "Structured Logging," "Appropriate Log Levels," and "Core-Aware Architecture." Each panel includes an icon and brief guidance (use JSON logs, set correct log levels, and apply retention/monitoring and billing policies).](https://mintcdn.com/kodekloud-c4ac6d9a/FoKJ_KwuI1oFSFo9/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/CloudWatch-Logs-and-Log-Insights-Beyond-the-Basics/best-practices-exam-tips-logging-architecture.jpg?fit=max\&auto=format\&n=FoKJ_KwuI1oFSFo9\&q=85\&s=7913e00ec6561c0ca306564784ebb861)

Key takeaways

* Hierarchy: log groups → log streams → log events.
* Retention is defined at the log group level.
* Metric filters convert new log events into CloudWatch metrics (limit 100 filters/group); metric generation does not add processing charges beyond normal ingestion and storage, but the resulting metrics may incur CloudWatch metric charges.
* Each log group supports only one subscription filter — use Kinesis Streams/Firehose or a Lambda forwarder to achieve multi-destination fan-out.
* Use selective retention, S3 archiving, sampling, and structured logging to control costs and improve query efficiency.
* Filter noise early and restrict Logs Insights queries to reduce scanned data and lower query costs.

Further reading and references

* [CloudWatch Logs Concepts (AWS Documentation)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html)
* [CloudWatch Logs Insights Query Syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
* [Kinesis Data Streams vs Kinesis Data Firehose](https://docs.aws.amazon.com/streams/latest/dev/what-is-kinesis.html)
* [Best practices for CloudWatch Logs security and retention](https://aws.amazon.com/blogs/mt/monitoring-aws-resources/)

That concludes this section on CloudWatch Logs and CloudWatch Logs Insights beyond the basics. Try hands-on exercises and scenario-based questions to apply these patterns.

> **[Watch Video](https://learn.kodekloud.com/user/courses/aws-certified-devops-engineer-professional/module/15ed9473-60f4-466a-b7b2-3d9cb829e847/lesson/ec7959ac-c78a-4828-bb84-aab5b2412c6e)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.