> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Subscription Filters and Log Streams to Kinesis Lambda and OpenSearch

> Designing CloudWatch Logs subscription pipelines to stream and transform logs to Kinesis, Lambda, Firehose or OpenSearch for real time processing, cross account aggregation, filtering, and analytics.

Welcome. This lesson explains how to use CloudWatch Logs subscription filters and log-streaming architectures to move logs beyond CloudWatch for real-time processing, analytics, compliance, and alerting.

When designing centralized logging across accounts and regions, the objective is to build a dynamic pipeline that collects, filters, routes, processes, and analyzes log events. This enables actionable insights, cross-service correlation, and enterprise-grade observability.

![A diagram titled "Moving Logs Beyond CloudWatch" showing CloudWatch Logs flowing through subscription filters and real-time processing into a dynamic pipeline that produces alerts, analytics, compliance, and BI.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/moving-logs-beyond-cloudwatch-pipeline.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=ef5a738e063cd02395019f121d7339be)

The pipeline diagram highlights common streamed-log consumers: BI/analytics, security tools, alerting, and compliance. CloudWatch Logs subscription filters provide near-real-time forwarding to downstream services so you can transform, enrich, and route logs where they’re most useful.

Key constraint: CloudWatch Logs permits only one active subscription filter per log group. You cannot attach multiple subscription filters to the same log group simultaneously — plan your architecture around this limitation.

![A slide diagram explaining the CloudWatch constraint "One Subscription Filter Per Log Group." The left (green) diagram shows a correct setup with one subscription filter routing to multiple destinations, and the right (red) diagram shows the incorrect use of multiple subscription filters.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/cloudwatch-one-filter-per-log-group.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=27d4e820fb5e0d2d4b53682e8e41a8be)

Plan your destination strategy early. Rather than creating many log groups to work around the single-filter constraint, route from a single subscription filter to a destination that can fan out (for example, Kinesis Data Streams) or perform transformations.

![A slide titled "The Critical Constraint You Must Know" showing a design flow: CloudWatch Logs -> Amazon Kinesis Data Streams -> AWS Lambda, with advice to "Plan destination strategy early" to avoid costly rework. Below the stream are icons for Analytics and Security Tools as downstream consumers.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/cloudwatch-kinesis-lambda-plan-destination.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=252b170fa1b7029dd6a8268e63ac4a3e)

Destination choices and trade-offs

| Destination | Use Cases | Pros | Considerations |
| - | - | - | - |
| Kinesis Data Streams | Real-time analytics, multiple independent consumers | High throughput, ordering guarantees, enhanced fan-out, independent consumers | Requires consumer logic and shard management |
| Lambda (subscription target) | Transformation, routing, enrichment, batching | Serverless, flexible and programmable | Watch concurrency, timeouts (max 15m), and retries |
| Amazon OpenSearch Service / OpenSearch Serverless | Full-text search, dashboards, SIEM-style analytics | Direct indexing and visualization | Often deployed via Firehose or Lambda to handle formatting and bulk indexing |
| Kinesis Data Firehose | Simple delivery to S3, Redshift, OpenSearch | Managed buffering, retries, transformations (via Lambda) | Less control than raw Kinesis streams, delivery semantics depend on destination |

Choose the destination that best supports your downstream consumers to avoid costly rework later.

![A slide titled "Three Destination Types" showing Amazon Kinesis Data Streams, AWS Lambda, and Amazon OpenSearch Service on the left. The right column outlines "Custom transformation and routing logic" with bullet points about serverless event-driven processing, transform/enrich/route logs, invocation patterns, and integrations.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/three-destination-types-kinesis-lambda-opensearch.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=533188807583f35be914fdef71955591)

Cross-account aggregation is essential for unified observability and centralized compliance. A common enterprise pattern uses a central logging account that hosts the primary stream or delivery endpoint (Kinesis stream, Firehose, or S3). Source accounts forward logs via subscription filters into that central destination.

Requirements for cross-account ingestion:

* Resource policies on the destination (e.g., Kinesis or Firehose) that allow the source account to put records.
* Cross-account IAM roles granting PutRecords/PutRecordBatch for Kinesis, or delivery permissions for Firehose/S3.
* Secure transport and least-privilege IAM policies to limit scope.

![A slide titled "Cross-Account Log Aggregation" showing a circular workflow around "Enterprise Environment." Three colored nodes highlight "Compliance-ready centralized AWS Logging" (blue), "Unified Observability" (green), and "Cost Management" (orange).](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/cross-account-aws-logging-observability-cost.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=6b094c4d39228dca13b37f5e8e3532e6)

Common implementation pattern

| Component | Purpose |
| - | - |
| Central logging account | Hosts Kinesis stream / Firehose / S3 for aggregation |
| Application accounts | Forward selected CloudWatch Logs via subscription filters |
| Cross-account IAM roles & policies | Allow source accounts to write to central destination |
| Downstream consumers | Archival, analytics, SIEM ingestion, dashboarding |

![A slide diagram titled "Cross-Account Log Aggregation" showing a common implementation pattern with four labeled components: Central Logging Account, 15+ Application Accounts, IAM Roles, and Aggregated Processing. Beneath those are three benefit boxes (centralizes compliance, reduces per-account costs, and enables org-wide log analytics).](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/cross-account-log-aggregation-diagram.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=daf2668994ebd92df0ea46ed85fba192)

Filter patterns let you reduce ingestion volume and focus on signal. Common strategies:

* Forward only ERROR or FATAL events.
* Extract and forward specific fields to minimize payload size.
* Use OR conditions to capture multiple critical levels.
* Use no filter (pass-through) when you require full fidelity—expect higher ingestion costs.

Example CloudWatch Logs filter patterns:

```text theme={null}
ERROR
[timestamp, request_id, level, message]
{ $.level = "ERROR" || $.level = "FATAL" }
```

Filter early to reduce downstream storage and processing costs. Use filtering for alert pipelines, fraud detection, security event capture, and cost optimization.

![The image is a slide titled "Filter Patterns for Targeted Processing" showing four strategic use cases arranged in a quadrant: 1) Alert pipelines, 2) Fraud detection, 3) Security events, and 4) Cost optimization, each with a colored corner graphic and icon. The slide is branded © Copyright KodeKloud.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/filter-patterns-targeted-processing-quadrant.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=988245219f3bf5953992131bdd5e046e)

OpenSearch integration options

* Direct to OpenSearch Service: If OpenSearch is the primary target, use Kinesis Data Firehose or a Lambda transformer to format and index logs directly into OpenSearch Service. This is common when you want real-time indexing and visualization.
* Lambda bridge to OpenSearch Serverless: When using OpenSearch Serverless (or when significant transformation is needed), insert a Lambda between the subscription and the OpenSearch API. Lambda receives batches, transforms/enriches records, and issues bulk index requests to the OpenSearch Serverless API. Also leverage OpenSearch tiered storage (hot/warm/cold) for cost/performance trade-offs.

![A presentation slide titled "OpenSearch Integration Patterns — Direct to OpenSearch Service" showing a colorful arrow-based flow diagram illustrating stages like built-in log ingestion, real-time indexing, and visualization-ready.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/opensearch-direct-integration-flow.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=e5b958fe0cb8d51ad241c9f467cc1fbf)

If you need serverless transformation before ingestion, use the Lambda bridge pattern. Lambda reads batches from the subscription, performs transformation and enrichment, batches records, and then issues bulk indexing operations to OpenSearch Serverless.

![A slide diagram titled "OpenSearch Integration Patterns" showing the "Lambda Bridge Pattern" for serverless log ingestion. It outlines four components—Subscription Filter → Lambda, Transform & Batch, Cost Optimization, and OpenSearch Serverless API—arranged around a central icon.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/opensearch-lambda-bridge-serverless-logs.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=b320db5d9803cb1a3615f16e2b2060dc)

Kinesis Data Streams: when to choose

Kinesis Data Streams is ideal when you need multiple independent consumers, high throughput, ordering guarantees, and flexible retention. Streaming from CloudWatch Log Groups into a Kinesis stream lets you support multiple downstream systems (alerting, archival, SIEM, analytics) concurrently.

![A presentation slide titled "Kinesis Data Streams Deep Dive" showing a "Multiple Consumer Architecture Pattern." It illustrates four consumers—Consumer 1: Real-Time Alerting, Consumer 2: Compliance Archival, Consumer 3: Fraud Detection, and Consumer 4: SIEM Integration—displayed as colored stepped blocks with icons.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/kinesis-data-streams-multiple-consumers-architecture.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=b9500a1263895bd4f7b6928695625dae)

Lambda considerations in log pipelines

When using Lambda for transformation, routing, or ingestion:

* Account for concurrency and provisioned concurrency needs.
* Implement robust error handling, DLQs (dead-letter queues), and retry strategies.
* Optimize memory and timeouts (Lambda max timeout is 15 minutes).
* Use durable buffers like SQS or Kinesis if downstream systems can cause backpressure or longer processing times.

![A presentation slide titled "Lambda Processing Patterns" showing production design considerations arranged as four numbered, color-coded boxes. The items listed include handling concurrent invocations, implementing robust error handling, optimizing function memory, and monitoring health metrics.](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/lambda-processing-patterns-design-considerations.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=1fbd3409987c029a0aa847c8319c8091)

Summary — must-know concepts

| Topic | Key Point |
| - | - |
| Subscription filters | Only one subscription filter per CloudWatch Log Group |
| Cross-account ingestion | Requires resource policies and IAM roles/permissions |
| OpenSearch Serverless | Typically uses a Lambda bridge for transformation and bulk indexing |
| Filter patterns | Use filters to reduce low-value volume and control costs |
| Kinesis Data Streams | Provides enhanced fan-out and multiple consumer support for high-throughput needs |

![A presentation slide titled "DOP-C02 Exam Focus Areas" showing five colorful numbered gear icons under the heading "Must-Know Concepts." Each gear lists a short point (e.g., one subscription filter per log group; cross-account IAM configuration; OpenSearch Serverless Lambda requirement; filter patterns reduce volume; Kinesis enhanced fan-out).](https://mintcdn.com/kodekloud-c4ac6d9a/SSy1aL2mVSdeEi1p/images/AWS-Certified-DevOps-Engineer-Professional/Domain-4-Monitoring-and-Logging/Subscription-Filters-and-Log-Streams-to-Kinesis-Lambda-and-OpenSearch/dop-c02-must-know-concepts-slide.jpg?fit=max\&auto=format\&n=SSy1aL2mVSdeEi1p\&q=85\&s=911476622fe6e51619957f9aec48ddff)

A hands-on demo typically follows to show configuration steps, IAM role setup, subscription filter creation, and end-to-end verification.

Further reading and references

* AWS CloudWatch Logs subscription filters: [https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html)
* Amazon Kinesis Data Streams: [https://docs.aws.amazon.com/streams/](https://docs.aws.amazon.com/streams/)
* Amazon Kinesis Data Firehose: [https://docs.aws.amazon.com/firehose/](https://docs.aws.amazon.com/firehose/)
* AWS Lambda developer guide: [https://docs.aws.amazon.com/lambda/](https://docs.aws.amazon.com/lambda/)
* Amazon OpenSearch Service: [https://docs.aws.amazon.com/opensearch-service/](https://docs.aws.amazon.com/opensearch-service/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/aws-certified-devops-engineer-professional/module/15ed9473-60f4-466a-b7b2-3d9cb829e847/lesson/f5656cba-f509-4200-9b1c-384b4533107b)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.