Demo Setting up Kafka on EC2

In this guide, you’ll deploy a single-node Apache Kafka broker on an AWS EC2 instance using KRaft mode (no ZooKeeper). The broker will serve as the event bus for your applications. You’ll complete the following steps:

1. Create an IAM Role for EC2 with SSM Access

1. In the AWS Console, go to IAM → Roles → Create role.
2. Choose AWS service → EC2, then click Next.
3. Search for SSM and attach the AmazonSSMFullAccess policy.
4. Name the role KafkaDemo, then Create role.

After attaching, the role’s trust policy will allow EC2 to communicate with Session Manager:

{
  "Principal": {
    "Service": ["ec2.amazonaws.com"]
  }
}

This enables secure shell-less access via the AWS console.

2. Launch the EC2 Instance

1. Open the EC2 Console → Launch instance.
2. Configure:
   • Name: kafka-demo
   • Instance type: t2.medium
   • Root volume: increase to 16 GiB
   • IAM role: KafkaDemo
3. Skip key pair selection (SSM will handle connectivity).
4. Keep the default security group for now.
5. Click Launch.

3. Connect via Session Manager

Wait until the instance state reads running. Then:

• Go to Instances, select kafka-demo → Connect → Session Manager → Connect.

4. Install Java and Kafka

In the Session Manager terminal, elevate privileges and install:

# Become root and navigate to home
sudo su
cd ~

# Download Kafka 3.0.0 for Scala 2.13
wget https://downloads.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
tar -xzf kafka_2.13-3.0.0.tgz
cd kafka_2.13-3.0.0

# Verify Java; install Corretto if missing
if ! command -v java &>/dev/null; then
  yum install -y java-1.8.0-amazon-corretto
fi

# Confirm installation
java -version

Expected output:

openjdk version "1.8.0_442"
OpenJDK Runtime Environment Corretto-8.442.06.1 (build 1.8.0_442-b06)
OpenJDK 64-Bit Server VM Corretto-8.442.06.1 (build 25.442-b06, mixed mode)

5. Configure Kafka in KRaft Mode

Kafka 3.x’s KRaft protocol removes the need for ZooKeeper. Perform these steps:

1. Generate a cluster ID

   bin/kafka-storage.sh random-uuid
   

   Copy the returned UUID (e.g., BMKCKvMMT64yxEZSmnTQ).

2. Format the storage directory

   bin/kafka-storage.sh format \
     -t BMKCKvMMT64yxEZSmnTQ \
     -c config/kraft/server.properties
   

3. Edit config/kraft/server.properties and update:

   # Roles and IDs
   process.roles=broker,controller
   node.id=1
   controller.quorum.voters=1@localhost:9093
   
   # Network listeners
   listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
   inter.broker.listener.name=PLAINTEXT
   
   # Replace with your instance’s public IP
   advertised.listeners=PLAINTEXT://<YOUR_EC2_PUBLIC_IP>:9092
   

6. Open Port 9092 in the Security Group

Allow external clients to reach Kafka’s default port:

1. In EC2 Console, select the instance → Security → Security groups.
2. Under Inbound rules, click Edit inbound rules → Add rule:
   • Type: Custom TCP
   • Port: 9092
   • Source: 0.0.0.0/0 (or restrict to your subnet)
   • Description: Kafka broker
3. Save rules.

7. Start Kafka and Create a Topic

Start the Kafka broker

bin/kafka-server-start.sh config/kraft/server.properties

Verify the console logs for successful startup.

Create a demo topic

Open a new Session Manager shell (keep the broker running):

sudo su
cd ~/kafka_2.13-3.0.0

bin/kafka-topics.sh --create \
  --topic cartevent \
  --bootstrap-server <YOUR_EC2_PUBLIC_IP>:9092 \
  --partitions 3 \
  --replication-factor 1

You should see:

Created topic cartevent.

Congratulations! Your single-node Kafka broker on EC2 is now online and ready to accept messages.

Links and References

• Apache Kafka Documentation
• Amazon EC2 User Guide
• AWS Systems Manager Session Manager