Skip to main content
Welcome back. In this lesson you’ll set up Apache Kafka on an EC2 instance. This single-node Kafka broker will act as the central message bus for our demo: front-end and back-end services will exchange events through the Kafka cluster running on this instance. High-level steps
  • Create an IAM role for the EC2 instance (to enable Session Manager).
  • Launch an EC2 instance and attach the IAM role.
  • Install Java and Apache Kafka.
  • Configure Kafka to run in KRaft mode (no ZooKeeper).
  • Open the Kafka broker port (9092) on the security group.
  • Start Kafka and create a topic for the demo.
Let’s begin in the AWS console. Create an IAM role for the EC2 instance and allow Session Manager access
  1. Open the IAM console → Roles → Create role.
  2. Choose EC2 as the trusted entity and continue.
The image shows an AWS IAM interface where a user is selecting a trusted entity type to create a role. Options include AWS service, AWS account, web identity, SAML 2.0 federation, and custom trust policy.
On the permissions page, attach the SSM policy that allows Session Manager access (for example AmazonSSMManagedInstanceCore). Give the role a descriptive name such as Kafka-demo and create it.
The image shows an AWS IAM management console screen where a role is being created. It highlights steps for adding permissions and tags, with a focus on the "AmazonSSMFullAccess" policy.
Session Manager lets you open a browser-based shell to your EC2 instance without SSH keys or open SSH ports. Attaching an IAM role with SSM permissions is the recommended approach for secure, keyless access.
Launch an EC2 instance and attach the IAM role
  • EC2 console → Launch Instance.
  • Name the instance (e.g., kafka-demo-broker).
  • Instance type: t2.medium.
  • If you plan to use Session Manager you may skip creating an SSH key pair.
  • Use the default security group for now (we’ll update it to allow Kafka traffic).
  • Increase the root volume from 8 GB to 16 GB.
  • Under Advanced Details → IAM instance profile, select the Kafka-demo role you created.
The image shows a screenshot of the AWS EC2 console where an instance is being configured, including storage options and security settings. It provides details about the free tier eligibility and summary of the instance configuration.
Launch the instance and wait until its state becomes running. Connect to the instance via Session Manager
  • Select the instance → Connect → Session Manager → Connect.
  • A browser shell opens and you can run commands directly as the EC2 user.
Prepare the instance (become root and set up working directory)
Install Kafka and Java
  1. Download and extract Kafka (example uses Kafka 3.0.0 for Scala 2.13):
  1. Check if Java is installed:
  1. Install OpenJDK 1.8 if needed:
  1. Verify the Java installation:
Configure Kafka to run in KRaft mode (ZooKeeperless) KRaft (Kafka Raft) mode lets Kafka manage metadata itself without ZooKeeper. The main steps are:
  • Generate a cluster ID and format storage for KRaft.
  • Update config/kraft/server.properties with KRaft-specific settings.
  • Start the Kafka server.
Generate the cluster ID and format KRaft storage
Edit the KRaft server properties Open config/kraft/server.properties (for example, with vim) and update the following key settings:
  • process.roles=broker,controller
  • node.id=1
  • controller.quorum.voters=1@localhost:9093
  • listeners should bind to 0.0.0.0 so Kafka accepts remote connections
  • advertised.listeners should use the EC2 public IP so external clients can connect
  • controller.listener.name=CONTROLLER and inter.broker.listener.name=PLAINTEXT
Example critical sections for config/kraft/server.properties:
Why listen on 0.0.0.0? Binding to 0.0.0.0 ensures the broker accepts connections from external network interfaces. If listeners bind only to loopback, external clients will be unable to connect. Open the Kafka port (security group) Edit the instance security group inbound rules and add a custom TCP rule for port 9092. For quick testing you may allow 0.0.0.0/0, but be cautious — restrict access in production to specific IP ranges, VPCs, or CIDR blocks.
The image shows an AWS console screen for editing inbound rules in a security group, with settings for allowing traffic from any IP address (0.0.0.0/0). There is also a warning about allowing access from all IP addresses.
Do not leave port 9092 open to the entire internet in production. Restrict access to trusted IP ranges, your VPC, or known CIDR blocks.
Start the Kafka broker (KRaft) Start Kafka in the foreground to watch logs while it initializes:
You should see logs indicating the Kafka Raft server and controllers started:
Create a demo topic Open a second Session Manager terminal (leave the broker terminal running) and create the topic cartevent:
Troubleshooting checklist Useful links and references Recap
  • You launched an EC2 instance with an IAM role that enables Session Manager access.
  • Installed Java and Kafka, formatted KRaft storage, and configured server.properties for KRaft mode.
  • Opened port 9092 and started the Kafka broker.
  • Created the cartevent topic ready for producers and consumers.
You’re now ready to configure your front-end service to produce events to the cartevent topic. See you in the next lesson.

Watch Video