Skip to main content
Hello and welcome back. In the previous lesson we learned about Kafka brokers and how they store incoming events inside a Kafka cluster. In this lesson we’ll focus on organizing the data that producers send into Kafka so consumers can discover and process the streams they need. Revisiting the previous example: brokers hold all events produced by our producers, but we still need a way to categorize those events so consumers can find the exact data they want. The logical container producers publish into is called a topic. Examples:
  • The EV charging station sends its events to a topic named EV_charging_topic.
  • Charging station metrics are published to a topic named station_metrics_topic.
A topic is a named stream that groups related messages. Unlike brokers — which are physical servers — topics are logical constructs built on top of brokers. When you create a Kafka cluster you provision brokers first, then create topics. Together, brokers and topics are the fundamental building blocks for producing and consuming events in Kafka. What is a topic?
  • A topic groups related messages logically. Think of each topic as a named stream for a particular type of data (device events, logs, user actions, metrics).
  • You can create many topics to organize data streams; Kafka itself doesn’t impose a strict upper limit, but practical constraints arise from cluster resources and metadata overhead. Tens of thousands of topics may require broker tuning.
Key features of Kafka topics
  1. Message categorization
  • Topics let you group similar messages so consumers subscribe only to needed data streams.
  • Proper topic design improves discoverability and simplifies downstream processing, analytics, and monitoring.
  1. Immutable, append-only logs
  • Messages in a topic are written to an append-only log. Once a record is written it cannot be modified.
  • This sequential, append-only storage preserves order within a partition (partitions are covered in the next lesson) and is important for use cases such as event sourcing and financial transactions.
  • Retention controls how long records are kept (for example, 7d or up to a certain size like 1GB). When retention thresholds are reached, older data is removed according to your configured policy.
Align retention settings with your application’s processing guarantees. If retention is too short, consumers might miss messages before they process them. If too long, you may incur unnecessary storage costs.
  1. Multi-consumer access
  • Multiple consumers (organized as consumer groups) can read from the same topic independently. Each consumer group maintains its own offsets (read positions), so different applications can consume the same data without interfering with one another.
  • This enables parallel analytics, monitoring, and real-time processing from a single event stream.
  1. Decoupled communication
  • Producers write to topics and consumers read from topics. Producers and consumers are decoupled and do not need to be aware of each other or be online simultaneously.
  • This decoupling supports scalable, asynchronous architectures where producers and consumers evolve independently.
  1. Replication (high availability)
  • Topics (more precisely, topic partitions) are replicated across multiple brokers to provide fault tolerance and data availability.
  • Replication ensures that if one broker fails, another broker holding a replica can continue to serve the data.
  • We will cover replication details and leader/follower behavior in a later lesson.
Comparison at a glance Why topic design matters (short checklist)
  • Keep related data together so consumers can subscribe to meaningful streams.
  • Consider retention: balance consumer needs vs storage costs.
  • Plan partitioning (next lesson) for throughput and ordering guarantees.
  • Use replication to protect against broker failures.
The image is an infographic about Kafka topics and their role in organizing data streams. It highlights five features: message categorization, immutable log, multi-consumer access, decoupled communication, and replication.
Those five aspects—categorization, immutability, multi-consumer access, decoupling, and replication—make Kafka topics the backbone of reliable real-time data streams. Topics provide structure and reliability for building event-driven architectures. Next lesson To understand how Kafka distributes and scales your data, we’ll dive into partitions and how they enable parallelism, ordering guarantees, and higher throughput. See you in the next lesson. Links and references

Watch Video