Demo Consumer Rebalancing

Welcome back. This practical demo shows Kafka consumer rebalancing in action. You’ll create a topic with multiple partitions, produce keyed messages so they land in specific partitions, and run multiple consumers in the same consumer group to observe how Kafka redistributes partition ownership as group membership changes. What we’ll do

Create a Kafka topic with 4 partitions
Produce keyed messages using a Python producer so messages map deterministically to partitions
Run Python consumers that join the same consumer group and observe rebalancing when consumers join/leave

Prerequisites

Kafka is installed and running on the host used in this demo (localhost:9092)
Python 3.8+ (or compatible) available

Quick commands

Task	Command / File
Create topic	`./kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4 --topic consumer-rebalancing-demo`
Create Python venv	`python3 -m venv kafka-demo-env`
Activate venv	`source kafka-demo-env/bin/activate`
Install client	`pip install kafka-python`
Producer script	`producer.py`
Consumer script	`consumer.py`

Useful references

1) Create the topic (4 partitions)

Create a topic named consumer-rebalancing-demo with 4 partitions and replication factor 1:

/root/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4 --topic consumer-rebalancing-demo

This registers the topic with the local Kafka broker running on localhost:9092.

2) Prepare a Python virtual environment and install kafka-python

Update packages and enable the Python venv module (if needed). Then create and activate a virtual environment and install kafka-python:

sudo apt update
sudo apt install -y python3.8-venv

# Create and activate the virtual environment
python3 -m venv kafka-demo-env
source kafka-demo-env/bin/activate

# Install kafka-python
pip install kafka-python

Expected installation output (truncated):

Collecting kafka-python
Downloading kafka_python-2.2.4-py2.py3-none-any.whl (308 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.2.4

A Python virtual environment keeps demo packages isolated from system Python packages. Activate the virtual environment (source kafka-demo-env/bin/activate) in every terminal you use for this demo.

3) Create the producer: producer.py

Create producer.py and paste the following code. This producer sends 1000 messages and cycles keys through four values so they map across the 4 partitions. Serializers handle None safely.

# producer.py
from kafka import KafkaProducer
import time
import random

# Initialize Kafka producer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    key_serializer=lambda k: k.encode('utf-8') if k is not None else None,
    value_serializer=lambda v: v.encode('utf-8')
)

topic = 'consumer-rebalancing-demo'

# Send 1000 messages to the topic
for i in range(1000):
    # Use a key that maps across 4 partitions (key distribution)
    key = f"key-{i % 4}"   # keys: key-0, key-1, key-2, key-3
    value = f"Message-{i}"
    producer.send(topic, key=key, value=value)
    print(f"Produced: {value} with key: {key}")
    time.sleep(0.5)  # Slow down message production for demo purposes

producer.close()

Run the producer:

python3 producer.py

You should see output like:

Produced: Message-0 with key: key-0
Produced: Message-1 with key: key-1
Produced: Message-2 with key: key-2
Produced: Message-3 with key: key-3
Produced: Message-4 with key: key-0
...

Because Kafka partitions messages deterministically based on the key (via the default partitioner), messages with the same key always go to the same partition. With keys cycling key-0..key-3 and 4 partitions, the messages will spread across partitions 0–3.

4) Create the consumer: consumer.py

Create consumer.py and paste the following script. It accepts an optional command-line argument for the consumer group id; if none is provided it uses default-group.

# consumer.py
from kafka import KafkaConsumer
import sys

def consume_messages(group_id):
    # Initialize Kafka consumer
    consumer = KafkaConsumer(
        'consumer-rebalancing-demo',
        bootstrap_servers=['localhost:9092'],
        group_id=group_id,
        auto_offset_reset='earliest',
        key_deserializer=lambda k: k.decode('utf-8') if k is not None else None,
        value_deserializer=lambda v: v.decode('utf-8') if v is not None else None
    )

    print(f"Consumer {group_id} is starting...")
    try:
        for message in consumer:
            print(
                f"Consumer {group_id} received: {message.value} "
                f"from partition {message.partition}, key: {message.key}"
            )
    except KeyboardInterrupt:
        print(f"Consumer {group_id} is stopping...")
    finally:
        consumer.close()

if __name__ == "__main__":
    group_id = sys.argv[1] if len(sys.argv) > 1 else "default-group"
    consume_messages(group_id)

Notes:

auto_offset_reset='earliest' ensures consumers without committed offsets will read from the beginning of the topic.
Kafka automatically creates consumer groups when a consumer first joins using the provided group_id. You do not need to pre-create the group.

5) Run consumers and observe rebalancing

Start one consumer in group consumer-group-1: Terminal 1

source kafka-demo-env/bin/activate
python3 consumer.py consumer-group-1

Example output (single consumer consuming from all partitions over time):

Consumer consumer-group-1 is starting...
Consumer consumer-group-1 received: Message-0 from partition 0, key: key-0
Consumer consumer-group-1 received: Message-1 from partition 1, key: key-1
Consumer consumer-group-1 received: Message-2 from partition 2, key: key-2
Consumer consumer-group-1 received: Message-3 from partition 3, key: key-3
...

With one consumer in the group, that consumer will be assigned all partitions (0, 1, 2, 3). Now open a second terminal and start another consumer joining the same group: Terminal 2

source kafka-demo-env/bin/activate
python3 consumer.py consumer-group-1

When the second consumer joins, Kafka triggers a group rebalance and redistributes partitions across group members. With 4 partitions and 2 consumers, a typical assignment is that each consumer receives two partitions (e.g., consumer A → partitions 0 & 1; consumer B → partitions 2 & 3). You will start seeing partition-specific messages in each terminal. Example after rebalance: Terminal A:

Consumer consumer-group-1 received: Message-100 from partition 0, key: key-0
Consumer consumer-group-1 received: Message-101 from partition 1, key: key-1
...

Terminal B:

Consumer consumer-group-1 received: Message-102 from partition 2, key: key-2
Consumer consumer-group-1 received: Message-103 from partition 3, key: key-3
...

If you start two more consumers (total 4 consumers in the same group), Kafka will rebalance again so each consumer receives one partition (1-to-1), enabling maximum parallelism:

Consumer 1 → partition 0
Consumer 2 → partition 1
Consumer 3 → partition 2
Consumer 4 → partition 3

Stopping a consumer — rebalance on leave

If you stop one consumer (Ctrl+C), Kafka removes it from the group and immediately triggers another rebalance.
The partitions previously owned by the stopped consumer will be reassigned to the remaining group members, who will begin receiving messages for those partitions.

This dynamic reassignment ensures each partition is consumed by exactly one consumer within a group at any time and is the core behavior behind Kafka consumer group rebalancing.

Consumer groups enable horizontal scaling of message consumption. The maximum number of active consumers that can consume in parallel in a group is bounded by the number of partitions. With N partitions, up to N consumers in the same group can be actively assigned partitions; additional consumers will remain idle until partitions are freed.

Summary

Created a topic with 4 partitions.
Produced keyed messages so they map deterministically to partitions.
Launched multiple consumers in the same group and observed Kafka rebalancing partition assignments as consumers joined or left.
With K partitions, up to K consumers can be actively assigned partitions; Kafka automatically rebalances when group membership changes.

That concludes this practical demo of Kafka consumer rebalancing.

Foundations of Event Streaming

Building Blocks of Kafka

Kafka Producers & Consumers: The Message Flow

Deep Dive into Kafka: Beyond the Basics

Confluent Kafka and Its Offerings

Kafka Connect: Effortless Data Pipelines

Project: Building an Event-Driven System

Introduction

Demo Consumer Rebalancing

1) Create the topic (4 partitions)

2) Prepare a Python virtual environment and install kafka-python

3) Create the producer: producer.py

4) Create the consumer: consumer.py

5) Run consumers and observe rebalancing

Summary

Watch Video

Practice Lab

​1) Create the topic (4 partitions)

​2) Prepare a Python virtual environment and install kafka-python

​3) Create the producer: producer.py

​4) Create the consumer: consumer.py

​5) Run consumers and observe rebalancing

​Summary

Watch Video

Practice Lab

1) Create the topic (4 partitions)

2) Prepare a Python virtual environment and install kafka-python

3) Create the producer: producer.py

4) Create the consumer: consumer.py

5) Run consumers and observe rebalancing

Summary