> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Streaming Data Sources Kinesis Flink Kafka > Comparing AWS Kinesis, Amazon Managed Flink, and Apache Kafka/MSK for building real-time streaming pipelines and ML feature processing, detailing components, trade-offs, and integration options. Traditional ML relies on static datasets, but many modern applications—fraud detection, anomaly detection, and personalized recommendations—require decisions as data arrives. Streaming data enables models and systems to adapt quickly, delivering low-latency predictions and insights that batch processing cannot provide. Streaming transforms The image compares two approaches to using streaming data for machine learning: training the model periodically with delayed predictions and updating the model incrementally for real-time predictions. static insights into continuous, real-time intelligence. This lets models stay current, react to new patterns, scale to high input volumes, and enable use cases that require immediate action. Overview: core streaming components * Streaming backbones — transport continuous event data (examples: Kinesis Data Streams, Kafka). * Processing engines — apply real-time logic, transformations, and feature engineering (examples: Flink, Lambda). * Delivery & integrations — reliably deliver streams to storage or analytics targets (example: Kinesis Data Firehose). * Analytics & consumers — dashboards, ML models, or downstream applications that consume processed streams. The image is an overview of a streaming data pipeline, detailing steps from ingestion and transport to processing, delivery, and analytics. It mentions tools and services like Kinesis, Kafka, Flink, and Lambda. Below is a focused look at AWS-native streaming options (Kinesis family and Flink on AWS) and alternative open-source options (Apache Kafka and Flink). Each plays a distinct role in real-time ML pipelines. AWS streaming services — components and roles Kinesis overview * Kinesis Data Streams and Kinesis Video Streams — ingest real-time telemetry, events, and media. * Kinesis Data Firehose — serverless delivery for loading streaming data into S3, Redshift, Amazon OpenSearch Service, and other sinks. * Kinesis Data Analytics — SQL-based stream processing for simple analytics and the ability to run Apache Flink applications for complex processing. The image is an infographic showing Amazon Kinesis, an AWS-native streaming service, with icons representing its components: Video Streams, Data Streams, and Data Firehose. Kinesis Data Firehose — serverless delivery Amazon Kinesis Data Firehose is a managed ETL-style delivery service that captures, transforms, buffers, and loads streaming data into data lakes and analytics stores with minimal operational overhead. Firehose supports lightweight transformations (including via AWS Lambda), format conversions (e.g., to Parquet/ORC), buffering and batching, and automated retries. The image is a diagram showing the data flow of Amazon Kinesis Data Firehose, with inputs leading into Amazon Kinesis and outputting to Amazon S3. Kinesis Data Streams — key characteristics * Real-time processing: ingest and analyze streaming data with minimal delay to enable timely detections and responses. * Scalability: scale throughput by increasing the number of shards. * Durability & availability: records are replicated across Availability Zones. * AWS integration: native connectors to S3, Redshift, SageMaker, Lambda, and more for downstream ML and analytics. * Custom applications: SDKs and client libraries enable bespoke consumer applications. The image lists five features of Kinesis: real-time data processing, scalability, data durability and availability, integration with AWS services, and custom application building. Typical AWS-native streaming pipeline * Producers (mobile apps, IoT devices, application logs) publish events to Kinesis Data Streams. * Consumers (AWS Lambda, Flink, or custom applications) process events in real time. * Kinesis Data Firehose handles delivery to storage and analytics with managed batching and retries. * Final sinks commonly include Amazon S3, Amazon Redshift, and Amazon OpenSearch Service. The image illustrates an AWS-native service data flow, showing how data moves from producers through Amazon Kinesis Data Streams to Amazon Kinesis Data Firehose, and finally into outputs like S3, Redshift, or OpenSearch. When to choose Amazon Kinesis * You want a fully managed, AWS-native streaming backbone with low operational overhead. * Your workload requires low-latency, high-throughput streaming on AWS. * Tight integration with AWS analytics and ML services (S3, Redshift, SageMaker, Lambda) is important. The image is a graphic explaining when to use Amazon Kinesis, highlighting three points: needing a fully managed AWS service, demanding low-latency high-throughput streaming, and wanting integration with other AWS services like S3, Redshift, and SageMaker. A single Kinesis shard supports roughly 1 MB/second of data write throughput and about 2 MB/second of data read throughput, plus up to 1,000 PUT records/second. Use these figures to plan shard counts based on your ingest and consumer patterns. Apache Flink — advanced stream processing Apache Flink is an open-source, distributed engine for stateful stream processing over unbounded data. On AWS, Amazon Managed Service for Apache Flink simplifies deployment, scaling, and operations. Flink is especially appropriate when you need advanced features such as event-time semantics, complex windowing, large stateful computations, and strong processing guarantees for ML feature engineering and analytics. The image is a flow diagram showing data processing using Amazon Managed Service for Apache Flink, with data moving from producers to Amazon Kinesis Data Streams, then to Apache Flink for real-time processing, and finally to output services like Amazon S3, Redshift, and OpenSearch. Amazon Managed Service for Apache Flink — strengths * Stream-first architecture: all data is treated as streams, enabling continuous processing. * Event-time processing: watermarks and event-time semantics handle out-of-order events. * Stateful computations: support for large application state across event streams for complex joins and aggregations. * Exactly-once consistency: strong guarantees when configured with state backends and checkpointing. * Rich connectors: integrate with Kafka, Kinesis, HDFS, Cassandra, Elasticsearch/OpenSearch, JDBC, and cloud storage. Apache Kafka and Amazon MSK Kafka is an open-source streaming platform focused on durability, replayability, and ordering semantics. It is a common choice for multi-cloud or hybrid architectures and for teams that depend on Kafka’s ecosystem. Amazon Managed Streaming for Apache Kafka (Amazon MSK) provides a managed Kafka control plane while preserving the Kafka-compatible APIs and ecosystem. Typical Kafka + Flink architecture * Producers push events to Kafka topics (MSK). * A stream processor (often Flink) consumes topics, applies business logic (filtering, aggregation, complex event patterns). * Processed results are written to sinks such as Amazon S3, Redshift, or OpenSearch, or forwarded to downstream apps. When to choose Kafka / Amazon MSK * You operate Kafka on-premises or across clouds and need compatibility. * You require explicit replay semantics, fine-grained partition control, or advanced ordering guarantees. * You want the flexibility of Kafka’s ecosystem but prefer delegating infrastructure management to AWS. High-level comparisons | Aspect | Amazon Kinesis | Apache Kafka / Amazon MSK | | ----------------- | ------------------------------------------------------------ | ----------------------------------------------------------------- | | Management | Fully managed AWS-native service | Managed option via Amazon MSK; still Kafka-compatible | | Ecosystem | Deep integration with AWS services (S3, Redshift, SageMaker) | Broader open-source ecosystem, many connectors and tools | | Replay & Ordering | Ordering per shard; replay within retention window | Robust replay via offsets; fine-grained partitioning and ordering | | Best fit | AWS-centric, serverless-focused pipelines | Multi-cloud/hybrid setups, Kafka-dependent ecosystems | Purpose-focused comparison | Purpose | Kinesis Data Streams | Managed Flink (Amazon) | Kafka / MSK | | ---------------------- | ------------------------------------------ | ---------------------------------------- | ----------------------------------------------- | | Ingestion backbone | Yes — AWS-native ingestion | No (processing engine) | Yes — widely used ingestion backbone | | Stream processing | Basic via Kinesis Data Analytics or Lambda | Primary — advanced stateful processing | Works with Flink, Spark, etc. for processing | | Exactly-once semantics | Limited compared to Flink | Strong exactly-once when configured | Depends on consumer/processing engine | | Integration | Native AWS integrations | Integrates with Kinesis, Kafka, S3, etc. | Wide connector support; cross-cloud flexibility | Summary This guide covered the roles and trade-offs for: * Streaming backbones: Amazon Kinesis Data Streams and Apache Kafka (Amazon MSK). * Processing engines: Apache Flink (Amazon Managed Service for Apache Flink) for advanced stateful processing. * Delivery services: Amazon Kinesis Data Firehose for serverless, reliable delivery to data lakes and analytics. Choosing between these options depends on operational preferences (fully managed AWS vs. open-source flexibility), required processing semantics (event-time, stateful computations, exactly-once), and the need for tight AWS integration. Links and references * [Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) * [Amazon Kinesis Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) * [Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/developerguide/what-is.html) * [Apache Kafka](https://kafka.apache.org/) * [Amazon MSK (Managed Streaming for Apache Kafka)](https://aws.amazon.com/msk/)