Skip to main content
Welcome to this hands-on guide for ingesting logs into Amazon Kinesis Data Streams. In this walkthrough you will:
  • Create a Kinesis Data Stream.
  • Grant CloudWatch Logs permission to write to the stream.
  • Configure a CloudWatch Logs subscription filter to forward log events to the stream for near‑real‑time processing.
This workflow is useful for building streaming analytics, ETL pipelines, or connecting logs to downstream consumers such as Kinesis Data Firehose, Lambda, or custom consumers. Create the Kinesis data stream
  • In the AWS Console, search for Kinesis and ensure “Kinesis Data Streams” is selected.
  • Click Create data stream and give it a name (for this lesson we’ll use LogStream).
A screenshot of the Amazon Web Services Kinesis console showing the "Amazon Kinesis services" header and the tagline about collecting, processing, and analyzing data streams. To the right is a "Get started" panel listing options like Kinesis Data Streams, Amazon Data Firehose, and Managed Apache Flink.
Choose capacity and distribution
  • For Data stream capacity choose either:
    • On-demand — no shard provisioning required; ideal for development and unpredictable workloads.
    • Provisioned — you specify shard count for predictable throughput and predictable cost.
  • For record distribution, choose:
    • User-specified (partition key) — preserves ordering by key (maps to shards).
    • Auto — balances records across shards; use this when ordering is not required.
Use the option that best matches your throughput and ordering needs. For many demos, On-demand + Auto is the simplest.
A screenshot of the AWS Management Console on the Amazon Kinesis "Create data stream" page, with the data stream name set to "LogStream." The page shows the "Data stream capacity" options including On-demand and Provisioned.
Record size and downstream considerations
  • The default maximum record size is 1024 KiB. If you increase this, verify every downstream consumer (Lambda, Firehose, custom consumers) can handle the larger records.
  • Large record sizes increase memory and throughput requirements; consider batching small log events into a single record instead of increasing the record size.
A screenshot of the Amazon Web Services console (Amazon Kinesis — Create data stream) showing the "Maximum record size" setting set to 1024 KiB. A warning box beneath advises verifying downstream applications can handle larger records.
Create the stream
  • Review your settings and create the stream.
  • Stream creation can take a few minutes (sometimes up to 10 minutes). When ready the stream status will be Active and you can view the configuration summary.
A screenshot of the AWS Management Console showing an Amazon Kinesis data stream named "LogStream" with a green banner indicating the stream was successfully created. The page displays a data stream summary (status Active, 1-day retention and other configuration details).
Give CloudWatch Logs permission to write to the stream CloudWatch Logs must assume a role and call the Kinesis API (for example kinesis:PutRecord) to deliver log events to your stream. You accomplish this by creating an IAM role with:
  1. A trust policy that allows the CloudWatch Logs service (logs.amazonaws.com) to assume the role.
  2. A permissions policy attached to the role that grants Kinesis write actions on your specific stream ARN.
Create a role in IAM
  • Navigate to IAM → Roles → Create role.
  • For the trusted entity, select AWS service and configure a trust policy that allows the CloudWatch Logs service to assume the role. Example trust policy:
Attach a permission (inline) policy
  • Attach (or create an inline) permissions policy that lets the role write to your Kinesis stream. Replace the ARN below with your stream’s ARN (copy it from the Kinesis stream details). Example permission policy:
  • Give the role a descriptive name, for example CloudWatchLogs-to-Kinesis-role, and finish creating it.
Quick reference — IAM role essentials:
Trust vs permission policies:
  • The trust policy controls which principal (service) can assume the role.
  • The permission (or inline) policy controls what the assumed role can do (for example, call kinesis:PutRecord).
Configure the CloudWatch Logs subscription filter
  • In the AWS Console go to CloudWatch → Log Management → Log groups.
  • Open the log group you want to stream and choose Subscription filters → Create Kinesis subscription filter.
A screenshot of the AWS CloudWatch console showing a log group named "LogGroup2" with the Actions menu and Subscription filters expanded (options like Create Kinesis subscription filter). The top shows navigation buttons such as View in Logs Insights, Start tailing, and Search log group.
When creating the subscription filter:
  • Destination: Choose the current account and select the Kinesis Data Stream (LogStream) you created.
  • Permissions: Select the role you created (CloudWatchLogs-to-Kinesis-role) instead of allowing CloudWatch to create a default role.
  • Distribution method: Choose how records are distributed across shards:
    • By log stream — preserves ordering per log stream.
    • Random — spreads records across shards without per-stream ordering. For log sources where ordering per source matters, choose “By log stream”.
A screenshot of the AWS CloudWatch console on the "Create Kinesis subscription filter" page showing the "Distribution method" section, with "By log stream" selected and "Random" listed as the other option.
Filter name and pattern
  • Provide a filter name (for example, KinesisSubscriptionFilter).
  • The filter pattern can be left blank to forward all log events. To forward only specific events, define a filter pattern.
  • Example log lines for reference:
After configuring the subscription filter, create it to begin streaming matching events to Kinesis. Verify the subscription
  • In the log group’s Subscription filters section you will see the subscription target and configuration.
  • When new log events arrive, CloudWatch evaluates them against the filter pattern and forwards matching events to your Kinesis stream using the IAM role you provided.
  • Monitor CloudWatch Logs and Kinesis metrics for delivery errors (e.g., throttling or permission denied).
Screenshot of the AWS CloudWatch console showing the "LogGroup2" log group details page. It displays the log group's ARN, creation time, subscription filters, and configuration options like custom field indexes and transformers.
Summary checklist
  • Create a Kinesis Data Stream (e.g., LogStream).
  • Create an IAM role with:
    • A trust policy allowing logs.amazonaws.com to assume the role.
    • A permission policy allowing kinesis:PutRecord on your stream ARN.
  • In CloudWatch Logs, create a subscription filter for your log group targeting the Kinesis stream and using your IAM role.
  • Leave the filter pattern blank to forward all events, or provide a pattern to filter. Choose distribution method according to ordering requirements.
Make sure the IAM role you select in the subscription is the role you created. If you allow CloudWatch to create a role automatically, it may not match the least-privilege permissions you intend.
Exam tip
  • CloudWatch Logs uses subscription filters to send events to Kinesis Data Streams. The role must be assumable by the CloudWatch Logs service (trust policy) and must have permission to call kinesis:PutRecord on the target stream (permissions policy).
Further reading and references

Watch Video