Explains Kafka brokers as durable, scalable servers for storing and serving event streams using an EV charging use case, covering replication, partitions, scalability, and operations
Welcome back. In this lesson we’ll drill into the core Kafka components that provide durable, scalable message storage: Kafka brokers and how they operate in a cluster. To make the concepts concrete, we’ll continue using our EV charging use case.Producers (for example, EV charging stations) publish events to a logical stream called a topic. Consumers (the app, analytics, billing, or monitoring services) read those topics to act on data. Where are these events stored? On Kafka brokers — the servers that persist and serve message data for the cluster.
Use case: charging station status reports
When a user searches for available chargers, the app must show free/occupied status for a given time window.
Each charging station (producer) emits status and availability events to a topic (for example, charger-status).
The authoritative, persisted stream of those events lives on Kafka brokers. Consumers read and materialize the current availability view as needed.
What is a Kafka broker?
A Kafka broker is the server process that stores topic data on disk, serves client requests for reads and writes, and coordinates with other brokers for replication and leadership. Brokers are the backbone of Kafka’s durability and throughput.
Key characteristics of Kafka brokers
Feature
What it does
Relevance to EV charging use case
Message management
Persists messages to disk, serves producer writes and consumer reads, enforces retention and partition ordering
Ensures status events are durably stored and can be replayed to rebuild state
Cluster node
Each broker is a node that participates in the distributed cluster
Multiple brokers provide capacity and redundancy for high availability
Partition placement & leadership
Topics are split into partitions; a leader broker handles reads/writes for each partition, followers act as replicas
Partitioning allows parallelism (throughput) for many chargers; leaders handle client traffic
Scalability
Add brokers to distribute partitions and load horizontally
Scale ingestion as more chargers or regions are added
Fault tolerance & durability
Replication across brokers protects against data loss; followers can be promoted if a leader fails
Maintains accurate charger state despite broker failure
Dynamic membership
Brokers can join/leave; the cluster controller reassigns leadership/replicas as needed
Enables rolling upgrades and elastic growth without service disruption
Retention vs durability: Kafka persists messages to disk and applies retention rules (time- or size-based). Replication across brokers provides durability — preventing data loss — while retention determines how long data remains queryable in the cluster.
Real-world scale and examples
Large companies run Kafka at vast scale to support real-time systems. Examples include:
Netflix — thousands of brokers across many clusters for event-driven streaming and personalization.
Pinterest — multi-trillion messages per day at very high throughput.
PayPal — large broker fleets to provide reliable transaction and event delivery.
LinkedIn — Kafka’s originator, operating large clusters for core real-time pipelines.
These deployments demonstrate how brokers deliver throughput, durability, and availability for streaming use cases — from payments to IoT and social platforms.
Summary
Kafka brokers are the durable storage and serving layer for event streams. In the EV charging scenario they store charger status events, enable replay and state reconstruction, and scale horizontally to support large fleets and high throughput.Links and references
