Skip to main content
Welcome to this hands-on demonstration on Apache Kafka. In this lesson, you will learn how to set up Apache Kafka as an event bus so that one program can publish messages while multiple programs consume them in real time. Kafka acts as a central hub where events from various producers are sent to designated topics. Consumers then subscribe to these topics to retrieve and process messages. In this demonstration, we will:
  • Create a simple Kafka environment using Docker Compose.
  • Develop a Python-based Kafka producer that generates messages.
  • Build a Kafka consumer to read and process those messages.

​
Environment Setup

First, update your system and install the necessary packages for Python 3 and virtual environments. Run the following commands in your KodeKloud playground labs terminal:
After installation, clear your screen and create a Python virtual environment to isolate all Kafka-related dependencies:
Creating a virtual environment helps prevent conflicts with system-wide packages and ensures a smooth dependency management experience.

​
Setting Up Kafka with Docker Compose

Next, configure Kafka and Zookeeper using Docker Compose. Create a file named docker-compose.yaml and paste the content below. This configuration uses the Zookeeper image (required for managing the Kafka cluster) and the Confluent Kafka image that relies on Zookeeper. Please note that a terminal view is provided in the image below for illustration. The file name and its content remain unchanged.
The image shows a terminal window with a new file named "python-kafka-producer.py" open, displaying a blank screen with tilde symbols on the left.
Save the file, then bring up the Kafka environment using:
This command pulls the necessary images and starts the containers for Zookeeper and Kafka in detached mode. Verify that the containers are running with:

​
Creating and Validating Kafka Topics

With the Kafka cluster running, proceed to list the available topics:
Since no topics exist initially, create a new topic named sample-topic:
Verify creation by listing the topics again:
For detailed information about the topic, use:
Adjust the number of partitions and the replication factor as needed. These parameters are critical for achieving higher throughput and ensuring fault tolerance in production environments.

​
Producing Messages with a Kafka Producer

Now, let’s create a Python script to produce sample events to our Kafka topic. Open a text editor (e.g., using vim) and create a file named python-kafka-producer.py. An optional terminal screenshot is shown below for visual reference. Follow the written instructions to enter the code.
The image shows a dark-themed terminal window with a text editor open, displaying a blank screen with a series of tilde (~) symbols on the left. The status bar at the bottom indicates the editor is in "INSERT" mode.
Paste the following code into the file:
Save the file and run the producer using:
As the producer runs, it continuously generates messages—with each message containing a timestamp and a value—and sends them to the Kafka topic.

​
Consuming Messages with a Kafka Consumer

In a separate terminal, activate the Python virtual environment and create a new file named python-kafka-consumer.py:
Paste the following code into the file:
Save the file and then run the consumer:
The consumer will subscribe to the sample-topic and begin printing any messages it receives from Kafka.

​
Demo Overview

In summary, this lesson demonstrated how to:
  • Set up a Kafka cluster using Docker Compose.
  • Create and validate Kafka topics.
  • Develop Python scripts for both producing and consuming messages.
The producer continuously sends messages that include a timestamp and a random value, while the consumer retrieves and prints these messages in real time. This setup shows how Kafka can serve as the central nervous system for data streaming applications, efficiently handling data ingestion and distribution. Happy coding and see you in the next lesson!

Watch Video

Practice Lab