Demo Kafka Setup with KRaft

Welcome to this walkthrough on setting up Apache Kafka in KRaft mode (no ZooKeeper) on CentOS. By the end, you’ll have a single-node Kafka cluster running with the built-in Raft quorum protocol.

Prerequisites

CentOS 7 or later
Java 11+ installed (java -version)
Minimum 4 GB RAM and 2 CPU cores

Download and Extract Apache Kafka

Download Kafka 3.0.0 (Scala 2.13):

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz

Extract the archive:
```
tar -xvf kafka_2.13-3.0.0.tgz
```
Enter the Kafka directory:
```
cd kafka_2.13-3.0.0
```

Explore the Directory Structure

List the top-level folders:

ls -l
# bin  config  libs  site-docs

Dive into the config directory and view the KRaft config files:

cd config
ls kraft
# controller.properties  server.properties

File	Purpose	Default Path
controller.properties	Configuration for the Raft controller (metadata)	`config/kraft/controller.properties`
server.properties	Broker settings + Raft metadata directories	`config/kraft/server.properties`

Note

You can customize log directories, ports, and listeners in these files before formatting storage.

Format Local Storage for KRaft

Before starting Kafka, initialize local storage and bind it to a cluster ID.

Generate a UUID for your cluster:

bin/kafka-storage.sh random-uuid
# Example output: fRbs-vkR9Uevh5Cwlwk

Format the storage with the generated UUID:
```
bin/kafka-storage.sh format \
  -t fRbs-vkR9Uevh5Cwlwk \
  -c config/kraft/server.properties
```
This embeds the cluster ID into the log directories (default: /tmp/kraft-logs).

Warning

Reformatting storage erases existing logs. Only run this command once on a fresh directory.

Start Kafka in KRaft Mode

Launch the combined broker and controller process:

bin/kafka-server-start.sh config/kraft/server.properties

Look for logs like:

2025-04-17 14:18:03,196 INFO [RaftManager nodeId=1] Completed transition to Leader(localId=1, epoch=1)
2025-04-17 14:18:06,711 INFO Kafka server 3.0.0-0 started (kafka.server.KafkaRaftServer)

If you see KafkaRaftServer and no zookeeper.connect references, your cluster is running in KRaft mode.

Verifying KRaft vs. ZooKeeper

KRaft mode logs mention RaftManager and KafkaRaftServer.
There are no ZooKeeper processes or settings.
The controller and broker run within a single JVM.

Next Steps

Create topics:

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Produce and consume messages:

bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic test --bootstrap-server localhost:9092 --from-beginning

Scale to multi-node KRaft clusters by repeating storage formatting with unique IDs and updating controller.quorum.voters.