# KodeKloud: Beta Courses Event Streaming with Kafka

## Event Streaming with Kafka

### Foundations of Event Streaming

- [What is Event Streaming](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/What-is-Event-Streaming/page.md): Defines event streaming, its roles and platforms like Apache Kafka, and real-time use cases such as taxi app flows, durable logs, ordering, replay, and stream processing
- [Introduction to Apache Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Introduction-to-Apache-Kafka/page.md): Introduction to Apache Kafka describing its role as a distributed event streaming platform, core concepts, architecture integration, and common use cases like stream processing and real-time analytics
- [Event Driven Architecture Basics](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Event-Driven-Architecture-Basics/page.md): Introduction to event-driven architecture and how Apache Kafka provides durable, scalable, decoupled event streams enabling persistence, replay, fault tolerance, and independent consumer scaling.
- [Kafka in Finance Real time Transaction Processing](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Kafka-in-Finance-Real-time-Transaction-Processing/page.md): Using Apache Kafka to enable real-time financial transaction processing, facilitating compliance, fraud detection, account updates, notifications, and auditable event streaming with scalable, low latency architecture
- [Use cases and Real World Applications Use Case 2](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Use-cases-and-Real-World-Applications-Use-Case-2/page.md): Explains using Apache Kafka to stream EV charging station status and session events for real time availability, billing, monitoring, and scalable stream processing
- [Demo Setting up Kafka Cluster and Kafka UI using Docker](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Demo-Setting-up-Kafka-Cluster-and-Kafka-UI-using-Docker/page.md): Guide to set up a local Kafka cluster with Docker and add a third party Kafka UI for exploring brokers, topics, and messages
- [Demo Creating First Topic in Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Foundations-of-Event-Streaming/Demo-Creating-First-Topic-in-Kafka/page.md): Guide to creating and configuring a Kafka topic using UI and CLI, explaining partitions, replication, retention, cleanup policies and best practices for development and production.

### Building Blocks of Kafka

- [Kafka Brokers The Foundation of Message Storage](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Building-Blocks-of-Kafka/Kafka-Brokers-The-Foundation-of-Message-Storage/page.md): Explains Kafka brokers as durable, scalable servers for storing and serving event streams using an EV charging use case, covering replication, partitions, scalability, and operations
- [Understanding Kafka Topics Organizing Your Data Streams](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Building-Blocks-of-Kafka/Understanding-Kafka-Topics-Organizing-Your-Data-Streams/page.md): Explains Kafka topics and features for organizing event streams, covering categorization, append only logs, retention, multi consumer access, decoupling, partitioning, and replication.
- [Kafka Partitions Scaling and Parallelizing Your Data Processing](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Building-Blocks-of-Kafka/Kafka-Partitions-Scaling-and-Parallelizing-Your-Data-Processing/page.md): Describes Kafka partitioning for distributing topic data across brokers to achieve parallelism, scalability, per key ordering, consumer group sizing, and trade offs when choosing partition counts
- [Kafka Replication Ensuring Data Reliability and Fault Tolerance](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Building-Blocks-of-Kafka/Kafka-Replication-Ensuring-Data-Reliability-and-Fault-Tolerance/page.md): Explains Kafka partition replication for durability, high availability, leader election, ISR management, configuration best practices, and failure recovery.
- [Demo Topics Partitions and Brokers](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Building-Blocks-of-Kafka/Demo-Topics-Partitions-and-Brokers/page.md): Guide to creating and inspecting Apache Kafka topics, configuring partitions and replication, and using Kafka CLI tools to view topic and broker metadata.

### Kafka Producers & Consumers: The Message Flow

- [What is a Kafka Producer](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/What-is-a-Kafka-Producer/page.md): Describes Kafka producers, message publishing and partitioning, essential configurations, tuning for reliability and throughput, plus a Java example and troubleshooting tips.
- [Demo Kafka Producers](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Demo-Kafka-Producers/page.md): Demonstrates creating a multi-partition Kafka topic and producing messages with a Python kafka-python producer, showing partitioning, message delivery, and Kafdrop inspection.
- [Serialization in Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Serialization-in-Kafka/page.md): Explains Kafka message serialization, common formats like JSON Avro Protobuf, configuring serializers and Schema Registry integration, and best practices for security, performance, and compatibility.
- [Understanding the Role of Message Keys](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Understanding-the-Role-of-Message-Keys/page.md): Explains how Kafka message keys determine partition assignment via hashing, ensuring per-key ordering, affecting parallelism and hotspots, and offers key design and sharding best practices.
- [Producer Acknowledgments Acks and Reliability Guarantees](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Producer-Acknowledgments-Acks-and-Reliability-Guarantees/page.md): Explains Kafka producer acknowledgment settings and their impact on durability, latency, throughput, and configuration best practices for preventing data loss.
- [What is a Kafka Consumer](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/What-is-a-Kafka-Consumer/page.md): Explains Kafka consumers how they read and process topic partitions, ordering guarantees, partition independence, replay and offset management, and the pull-based consumption model.
- [Demo Kafka Consumers](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Demo-Kafka-Consumers/page.md): Guide to creating Kafka topics and writing Python producers and consumers with kafka-python, covering setup, message production, consumption, partitions, offsets, and production considerations
- [Consumer Groups and How They Work](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Consumer-Groups-and-How-They-Work/page.md): Explains Kafka consumer groups, partition assignment, scaling, fault tolerance, rebalancing, ordering guarantees, delivery semantics, and best practices for scalable, coordinated consumption of topic data.
- [Consumer Rebalancing](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Consumer-Rebalancing/page.md): Explains Kafka consumer rebalancing, triggers, eager vs cooperative protocols, pauses during reassignment, and operational best practices
- [Demo Consumer Rebalancing](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Demo-Consumer-Rebalancing/page.md): Demo showing how to create a multi-partition Kafka topic, produce keyed messages, and run consumers in a group to observe automatic partition rebalancing as consumers join and leave.
- [Section Recap](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Producers-Consumers-The-Message-Flow/Section-Recap/page.md): Summary of Kafka concepts including producers, partitions, consumers and groups, storage and rebalancing; demos and upcoming topics on replication, log management, and performance tuning.

### Deep Dive into Kafka: Beyond the Basics

- [Offset Management in Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Offset-Management-in-Kafka/page.md): Explains Kafka offset management, how consumers track and commit offsets, commit strategies, storage, and best practices to ensure correct, reliable message processing and avoid duplicates or data loss.
- [Poison Pill in Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Poison-Pill-in-Kafka/page.md): Explains Kafka poison pill problem where a malformed message stalls partition processing and outlines causes, real world impact, and mitigation patterns like DLQs and safe consumer handling.
- [Mitigation Strategies Handling Poison Pill](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Mitigation-Strategies-Handling-Poison-Pill/page.md): Guide for detecting, isolating, and recovering from malformed Kafka messages using schema enforcement, dead letter queues, bounded retries, and upstream filtering or transformation
- [ZooKeepers Role in Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/ZooKeepers-Role-in-Kafka/page.md): Explains ZooKeeper's historical role in Kafka cluster coordination, covering broker registration, controller election, metadata storage, notifications, offset history, operational risks, and KRaft replacement.
- [Demo Kafka Setup with Zookeeper](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Demo-Kafka-Setup-with-Zookeeper/page.md): Guide to installing and running Apache Kafka with ZooKeeper on CentOS, showing download, configuration, starting services, and verifying Kafka connects to ZooKeeper
- [Kafka KRaft](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Kafka-KRaft/page.md): Explains Kafka's KRaft architecture replacing ZooKeeper with embedded Raft consensus to simplify operations, improve scalability, and streamline migrations.
- [KRaft in Action New Broker joining Kafka Cluster](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/KRaft-in-Action-New-Broker-joining-Kafka-Cluster/page.md): Explains how a new Kafka broker joins a KRaft cluster via controller leader registration, Raft metadata log commits, and subsequent metadata updates and rebalancing.
- [Demo Kafka Setup with KRaft](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Demo-Kafka-Setup-with-KRaft/page.md): Step-by-step guide to setting up Apache Kafka in KRaft mode on CentOS including config inspection, generating cluster ID, formatting metadata storage, starting broker, and verifying logs.
- [Kafka Security](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Kafka-Security/page.md): Overview of Kafka security best practices using a banking example, covering encryption in transit and at rest, authentication and authorization, schema use, and access controls for sensitive event data.
- [Securing Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Securing-Kafka/page.md): Guide to securing Kafka clusters by implementing authentication methods, enforcing ACLs, and protecting data at rest with encryption and best practices.
- [Section Recap](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Deep-Dive-into-Kafka-Beyond-the-Basics/Section-Recap/page.md): Summary of Kafka best practices covering offset management, handling poison pill messages, ZooKeeper versus KRaft architecture, and securing Kafka with TLS SASL and ACLs.

### Confluent Kafka and Its Offerings

- [Beyond Installation The True Complexity of Managing Kafka](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Confluent-Kafka-and-Its-Offerings/Beyond-Installation-The-True-Complexity-of-Managing-Kafka/page.md): Explains operational challenges of running Apache Kafka in production and compares self-managed versus managed options to reduce complexity, covering scaling, security, observability, and integration
- [Confluent Cloud Kafka Reimagined for the Cloud Era](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Confluent-Kafka-and-Its-Offerings/Confluent-Cloud-Kafka-Reimagined-for-the-Cloud-Era/page.md): Overview of Confluent Cloud, a managed, cloud-native Kafka platform offering elastic scaling, tiered storage, high availability, security, schema management, connectors, and stream processing for production event streaming.
- [Demo Getting Started with Confluent for Free](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Confluent-Kafka-and-Its-Offerings/Demo-Getting-Started-with-Confluent-for-Free/page.md): Guide to creating a free Confluent Cloud account, provisioning a Kafka cluster, creating topics, and producing and inspecting sample event data.

### Kafka Connect: Effortless Data Pipelines

- [Streaming data from Kafka to other systems](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Connect-Effortless-Data-Pipelines/Streaming-data-from-Kafka-to-other-systems/page.md): Guide to Kafka Connect for building scalable fault tolerant pipelines that stream Kafka topics into long term stores like S3 data warehouses and databases
- [Demo Kafka Connect Setting up Kafka using KRaft](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Connect-Effortless-Data-Pipelines/Demo-Kafka-Connect-Setting-up-Kafka-using-KRaft/page.md): Guide to provisioning an EC2 instance and setting up a single-node KRaft Kafka broker for Kafka Connect and S3 integration
- [Demo Setting up S3 Kafka Connect](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Connect-Effortless-Data-Pipelines/Demo-Setting-up-S3-Kafka-Connect/page.md): Step-by-step guide to configure and run Kafka Connect standalone with Confluent S3 Sink connector to stream Kafka topic data into an S3 bucket, including IAM and config steps.
- [Demo Kafka Connect in Action](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Kafka-Connect-Effortless-Data-Pipelines/Demo-Kafka-Connect-in-Action/page.md): Demonstration of producing Kafka events to a topic and using Kafka Connect S3 sink to write those events as JSON files into an S3 bucket.

### Project: Building an Event-Driven System 

- [Architecture Overview of Event Driven System](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Project-Building-an-Event-Driven-System/Architecture-Overview-of-Event-Driven-System/page.md): Overview of building an end-to-end event-driven e-commerce architecture using Apache Kafka, mapping UI events to producers, brokers, and independent consumers with a near-real-time dashboard
- [Demo Setting up Kafka on EC2](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Project-Building-an-Event-Driven-System/Demo-Setting-up-Kafka-on-EC2/page.md): Guide to deploy a single-node Apache Kafka broker on an AWS EC2 instance using KRaft, Session Manager access, IAM role setup, security group configuration, and creating a demo topic
- [Demo Starting Our Frontend Interface](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Project-Building-an-Event-Driven-System/Demo-Starting-Our-Frontend-Interface/page.md): Guide to building a Flask Toy Shop frontend that produces JSON order events to a Kafka cartevent topic, runs locally, and shows how to verify messages on the broker.
- [Demo Starting Our Internal Interface](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Project-Building-an-Event-Driven-System/Demo-Starting-Our-Internal-Interface/page.md): Setting up a warehouse backend UI that consumes front-end order events from Kafka, configures a consumer, and demonstrates running and testing the packer dashboard.
- [Section Recap](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Project-Building-an-Event-Driven-System/Section-Recap/page.md): Recap of building an event driven order processing system using Apache Kafka where front end producers send events to topics and consumers update a warehouse dashboard

### Introduction

- [Course Introduction](https://notes.kodekloud.com/docs/Event-Streaming-with-Kafka/Introduction/Course-Introduction/page.md): Introduction to Apache Kafka event streaming, covering core concepts, producers and consumers, partitioning strategies, hands-on labs, Docker sandbox, and cloud integration with IAM