Skip to main content
Welcome back. This lesson builds on the fundamentals of Apache Kafka and clarifies the event-driven architecture (EDA) problem we aim to solve. We’ll revisit the basic event flow, examine common failure modes at scale, and explain how Kafka changes the architecture to provide durability, scalability, and decoupling. Overview of a simple event flow
  • System A emits an event. A producer can be a microservice, an IoT device, a mobile app, or any application component.
  • System B consumes and processes that event, potentially producing output that System X will later consume.
  • After processing, System B may acknowledge the event, notify System A, or persist the result.
This cycle—produce, consume, process, acknowledge—is straightforward at small scale. When System A emits n events concurrently, System B must process all n events without losing any and often must reply or persist the results. At scale, throughput, persistence, failure handling, and response coordination become challenging.
The image illustrates the basics of event-driven architecture, showing a flow of events and responses between System A, System B, and System X. Events are issued by System A and processed by System B, which interacts with System X, returning responses back to System A.
Why this becomes hard at scale
  • High concurrency increases load and exposes transient failures.
  • Synchronous point-to-point integrations create tight coupling and deployment friction.
  • Lack of durable storage for events can lead to data loss and inconsistent state.
Apache Kafka provides a durable, distributed event log that decouples producers and consumers, allowing each to scale and evolve independently. It changes the contract: once an event is durably written to Kafka, the producer’s responsibility ends and consumers can process at their own pace.
The image illustrates the basics of event-driven architecture, showing how events from "System A" are sent to "System B" and how responses are managed, with connections to "System X" and a processed event indicator.
Common pitfalls in event-driven architectures
The image outlines pitfalls of event-driven architecture, including tight coupling, reduced scalability, single points of failure, and no message persistence.
The practical consequence: without durable persistence and replay, a logistics tracking system can lose status updates; a recommendation engine may stop serving users when its pipeline fails.
Kafka addresses many of these pitfalls by providing a durable event log that decouples producers and consumers, supports high throughput, enables replay, and offers built-in fault tolerance and scalability.
How Kafka transforms the architecture
  • Producers (System A) write events to Kafka topics and return immediately after the event is durably stored; producers do not need to manage downstream state.
  • Multiple independent consumers (System B, System X, etc.) can read the same events at their own pace, maintain offsets, and reprocess when needed.
  • Consumers scale independently; Kafka partitions and consumer groups enable parallel consumption without changing producers.
  • Persistence, replication, and retention policies allow replay, auditing, and backfills for analytics and error recovery.
This decoupling simplifies deployments, reduces coordination overhead, and adds operational resilience.
The image depicts a diagram illustrating the role of Kafka as a message broker, connecting systems A, X, and B, resulting in a processed event.
Kafka’s core strengths
The image describes Kafka as the backbone of event-driven architectures, highlighting its features like high throughput, fault tolerance, scalability, and real-time processing capabilities.
Design considerations and practical tips
  • Model events as facts: use immutable, append-only events for reliable replay and auditability.
  • Choose partition keys that balance throughput and ordering requirements.
  • Tune retention and compaction policies based on recovery and storage needs.
  • Use consumer groups to scale processing while preserving partition-level ordering when required.
Design choices (partitioning, retention, ordering guarantees) directly affect scalability and correctness. Test failure scenarios and recovery flows to ensure your pipeline meets SLAs.
Conclusion With a durable event log like Kafka, systems can react to events as they occur, replay historical events when needed, and evolve independently without tight coupling. In the next lesson we’ll dive into Kafka’s core primitives—topics, partitions, producers, consumers, and consumer groups—and walk through concrete patterns for building resilient, scalable event-driven systems. Links and references

Watch Video