Skip to main content
Hello and welcome to this Learn-By-Doing course for Apache Kafka. My name is Vijin Palazhi, and I will guide you through the course content and hands-on labs. This practical course — created by Rakshith — is tailored for DevOps engineers, system administrators, and cloud practitioners who want a hands-on path to mastering Apache Kafka: a distributed event streaming platform frequently used to build real-time data pipelines and streaming applications. In this lesson/article you will gain practical experience with Kafka’s core concepts and components, including producers, consumers, topics, partitions, and brokers. You will learn how to set up a local Kafka environment, create and configure topics, and build simple producer and consumer applications to send and receive messages. You will also explore Kafka’s architecture and how producers and consumers interact with the cluster to process streaming events. The course walks you from fundamentals to practical development tasks and deployment patterns. By the end of the course, you will have a solid understanding
The image features the Apache Kafka logo at the top, with three icons below labeled "Setting Up," "Developing," and "Understanding."
of how to work with Apache Kafka and leverage it to handle streaming data efficiently within your organization. Quick navigation tips for the hands-on labs
  • Overview tab: Read this first. It contains concept summaries, architecture diagrams, and any prerequisites.
  • Tasks tab: Follow the numbered, step-by-step workbook to complete the lab exercises.
  • Terminal: Run all commands in the lab terminal. Use Hint and Solution tabs if you need help. Click Check to validate your work and progress.
The terminal provided in each lab is where you will run all hands-on commands. If you get stuck, use the Hint and Solution tabs, and click the Check button to validate your solutions and progress.
Example build command you might encounter when working with local web projects (run in the lab terminal):
What you will learn (high level)
  • Introduction to Apache Kafka: what it is, how it works, and why it’s used for real-time streaming.
  • Kafka architecture and components: brokers, ZooKeeper (or KRaft in newer releases), producers, consumers, topics, and partitions.
  • Producer and consumer development: send and receive messages, and build simple apps.
  • Topics and partitions: how partitioning enables parallelism and horizontal scaling.
  • Environment setup: configuring brokers, topics, and client settings.
  • Hands-on exercises: build and run producer and consumer applications to verify message flow.
Core components at a glance
Important: Kafka’s architecture has evolved. Older releases use ZooKeeper for metadata management; newer releases offer KRaft (Kafka Raft) mode to eliminate ZooKeeper. Check the Kafka release notes and lab prerequisites for the target version you’ll use in each exercise.
Course structure and learning path
  • Lab 1 (Foundations): Set up a local Kafka environment, create topics, and run basic producer/consumer examples.
  • Lab 2 (Producers): Build simple producer applications and explore key serializers, partitioning, and batching.
  • Lab 3 (Consumers): Implement consumers, manage offsets, and practice consumer groups and rebalancing.
  • Lab 4 (Operations): Configure broker settings, monitor cluster health, and perform simple troubleshooting.
  • Labs conclude with a mini-project that ties together producing, consuming, and topic management.
Links and references Estimated time and prerequisites
  • Estimated time: 3–6 hours total, depending on familiarity with distributed systems and command-line tools.
  • Prerequisites: Basic Linux command-line skills, familiarity with Java (or a client language), and a working web terminal (provided in labs).
Ready to get started? Start by reviewing the Overview for Lab 1 to learn the necessary concepts before attempting the Tasks.

Watch Video