Demo Topics Partitions and Brokers

Hello, and welcome back. In this lesson we’ll practice creating and inspecting Apache Kafka topics, configuring partitions and replication, and using the Kafka CLI tools to examine topic and broker metadata.

Lab environment

This lab runs in the KodeKloud environment where Kafka is already installed. Start by navigating to the Kafka installation directory to locate the binaries you’ll use:

# change to the Kafka installation directory
cd /root/kafka

# list the top-level files
ls -l

The Kafka command-line scripts live in the bin directory. Change into bin to see helper scripts used for managing Kafka:

cd /root/kafka/bin
ls -l

Example excerpt (truncated):

-rwxr-xr-x 1 root root   863 Sep 13  2022 kafka-topics.sh
-rwxr-xr-x 1 root root   895 Sep 13  2022 kafka-console-producer.sh
-rwxr-xr-x 1 root root   723 Sep 13  2022 kafka-console-consumer.sh
-rwxr-xr-x 1 root root  1010 Sep 13  2022 kafka-server-start.sh
...

Quick reference: common Kafka CLI commands

Purpose	Command example
Create or manage topics	`./kafka-topics.sh --create --topic <name> --bootstrap-server localhost:9092`
Describe topic metadata	`./kafka-topics.sh --describe --topic <name> --bootstrap-server localhost:9092`
Change topic config	`./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name <name> --alter --add-config retention.ms=...`
Broker API versions & list	`./kafka-broker-api-versions.sh --bootstrap-server localhost:9092`

Create a simple topic

Create a topic named demo_topic with the Kafka topics script:

# create a topic named demo_topic
./kafka-topics.sh --create --topic demo_topic --bootstrap-server localhost:9092

What the flags mean:

--create — create a new topic.
--topic demo_topic — the topic name.
--bootstrap-server localhost:9092 — address of a broker to bootstrap against (replace with your cluster’s bootstrap server(s)).

Typical response:

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic demo_topic.

You can also verify the newly created topic using a web UI such as Kafdrop or Confluent Control Center.

The image is a screenshot of a Kafdrop interface showing a Kafka Cluster Overview, including details about bootstrap servers, brokers, and topics.

Clicking the topic in the UI will display configuration and partition metadata.

Create a topic with partitions and replication

Partitions increase throughput and parallelism by spreading data across brokers. Create a topic with three partitions and a replication factor of 1:

./kafka-topics.sh --create --topic partitioned_topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

Best practices:

Align partition counts to your throughput and consumer parallelism needs. More partitions can improve parallelism but add overhead for management, disk usage, and leader elections.

Replication factor must not exceed the number of available brokers. If you set --replication-factor greater than your broker count, topic creation will fail.

Describe a topic (inspect metadata)

To view topic metadata—partition count, leader, replicas, in-sync replicas (ISR), and per-topic configs—use the --describe option:

./kafka-topics.sh --describe --topic partitioned_topic --bootstrap-server localhost:9092

Sample output:

Topic: partitioned_topic    TopicId: X8KJGSg5hThQHfo3DTm3eg    PartitionCount: 3    ReplicationFactor: 1    Configs: segment.bytes=1073741824
    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Partition: 1    Leader: 1    Replicas: 1    Isr: 1
    Partition: 2    Leader: 1    Replicas: 1    Isr: 1

Change topic configuration (retention example)

Topic-level configurations can be updated with kafka-configs.sh. For example, set retention to 2 days (milliseconds):

# set retention to 2 days (in milliseconds)
./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name partitioned_topic --alter --add-config retention.ms=172800000

Expected response:

Completed updating config for topic partitioned_topic.

Verify the change by re-describing the topic:

./kafka-topics.sh --describe --topic partitioned_topic --bootstrap-server localhost:9092

Sample updated output showing retention.ms:

Topic: partitioned_topic    TopicId: X8KJGSg5hThQHfo3DTm3eg    PartitionCount: 3    ReplicationFactor: 1    Configs: segment.bytes=1073741824,retention.ms=172800000
    Partition: 0    Leader: 1    Replicas: 1    Isr: 1
    Partition: 1    Leader: 1    Replicas: 1    Isr: 1
    Partition: 2    Leader: 1    Replicas: 1    Isr: 1

Inspect broker API versions and broker list

To confirm broker identities and supported API versions (useful when troubleshooting compatibility or determining available features), run:

./kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Sample output (truncated):

kafka-node:9092 (id: 1 rack: null) ->
    Produce(0): 0 to 9 [usable: 9],
    Fetch(1): 0 to 13 [usable: 13],
    ListOffsets(2): 0 to 7 [usable: 7],
    Metadata(3): 0 to 12 [usable: 12],
    LeaderAndIsr(4): 0 to 6 [usable: 6],
    ...
    CreatePartitions(37): 0 to 3 [usable: 3],
    DescribeCluster(60): 0 [usable: 0]

In this lab you have a single broker (sufficient for practicing topic creation and configuration changes). In production, use at least three brokers for fault tolerance and to support replication.

Foundations of Event Streaming

Building Blocks of Kafka

Kafka Producers & Consumers: The Message Flow

Deep Dive into Kafka: Beyond the Basics

Confluent Kafka and Its Offerings

Kafka Connect: Effortless Data Pipelines

Project: Building an Event-Driven System

Introduction

Demo Topics Partitions and Brokers

Lab environment

Quick reference: common Kafka CLI commands

Create a simple topic

Create a topic with partitions and replication

Describe a topic (inspect metadata)

Change topic configuration (retention example)

Inspect broker API versions and broker list

Further reading and resources

Watch Video

​Lab environment

​Quick reference: common Kafka CLI commands

​Create a simple topic

​Create a topic with partitions and replication

​Describe a topic (inspect metadata)

​Change topic configuration (retention example)

​Inspect broker API versions and broker list

​Further reading and resources

Watch Video

Lab environment

Quick reference: common Kafka CLI commands

Create a simple topic

Create a topic with partitions and replication

Describe a topic (inspect metadata)

Change topic configuration (retention example)

Inspect broker API versions and broker list

Further reading and resources