> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Notifications That Survive
> Lab demonstrating fan-out notification delivery to push and email workers, showing duplicated and redelivered messages, worker retries, and dead-letter queue handling for failed notifications
This lab runs a fan-out notification pipeline where a single published event is delivered to two independent workers — a push worker and an email worker — each consuming from its own queue. During the exercise you will observe:
* A single event duplicated across multiple queues (fan-out delivery).
* Redelivery and duplicate notifications (at-least-once semantics and crash/retry behavior).
* Undeliverable or repeatedly failing notifications routed to a dead-letter queue (DLQ) for inspection or replay.
Before starting, confirm that your message broker and worker processes are running and that the push, email, and dead-letter queues (and their bindings) are configured. If you use a hosted broker or a local instance, verify connectivity and broker admin UI access first.
## Prerequisites
* Running message broker (e.g., RabbitMQ, Kafka, or another AMQP/streaming system).
* Push and email worker processes that consume from separate queues.
* Dead-letter queue configured and bound according to your broker's DLQ policy.
* Access to broker management UI or CLI to inspect queue depths and message metadata.
## What you will learn
* How a single published event is duplicated across multiple queues so different workers can handle different delivery channels.
* Why duplicate deliveries occur (for example, at-least-once delivery, worker crashes, explicit negative acknowledgements).
* How a dead-letter queue collects messages that cannot be processed after configured attempts or are explicitly rejected.
## High-level flow
1. A producer publishes an event to a fan-out exchange or routing mechanism.
2. The exchange routes a copy of the event to both the push queue and the email queue (fan-out).
3. Each worker consumes messages from its queue and attempts delivery:
* On successful delivery the worker acknowledges (ACK) and the broker removes the message from that queue.
* If a worker crashes or NACKs with requeue, the broker may redeliver the message, producing duplicates.
* When a message exceeds the retry threshold or is explicitly rejected, the broker moves it to the dead-letter queue (DLQ).
4. The DLQ stores failed messages for inspection, debugging, or replay.
## Key components and where to check
| Component | Purpose | What to inspect |
| - | - | - |
| Producer | Publishes events to the exchange | Publish logs and producer retries |
| Exchange / Routing | Fan-out distribution to queues | Exchange type (fanout/topic), bindings, routing keys |
| Push queue & Email queue | Each holds messages for respective workers | Queue depth, `redelivered` flag, delivery count |
| Workers (push/email) | Consume and deliver notifications | Worker logs, ACK/NACK behavior, crash/retry patterns |
| Dead-letter queue (DLQ) | Stores undeliverable messages | DLQ contents, headers indicating original queue and delivery attempts |
## Example message payload
```json theme={null}
{
"event_type": "user.signup",
"user_id": "12345",
"email": "user@example.com",
"timestamp": "2026-10-07T12:34:56Z"
}
```
Keep the JSON payload and any headers (such as `x-death` in RabbitMQ) available when debugging redeliveries and DLQ routing.
## What to inspect during the lab
* Producer/publish log to confirm the event emission.
* Push worker logs and email worker logs to verify each receives and processes the event.
* Broker queue metadata:
* Queue depths (ready vs unacked).
* Message properties / headers (e.g., `redelivered`, delivery count, `x-death`).
* Dead-letter queue for messages that failed permanently and for diagnostic headers.
## Troubleshooting tips
* Only one worker receives the event:
* Verify the exchange type (fanout vs direct/topic).
* Confirm both queues are bound to the exchange with the correct routing keys.
* Seeing duplicates:
* Check worker stability and crash/restart logs.
* Inspect the broker's delivery semantics (at-least-once vs exactly-once capabilities).
* Confirm the worker acknowledges messages only after successful processing.
* Messages in DLQ unexpectedly:
* Review DLQ policy settings (max delivery attempts, TTL, rejection rules).
* Check whether workers are rejecting messages or letting them timeout.
* No messages published:
* Validate producer connection, authentication, and publish errors or retries.
## Common broker troubleshooting commands and links
* RabbitMQ management UI: [https://www.rabbitmq.com/management.html](https://www.rabbitmq.com/management.html)
* RabbitMQ dead-lettering: [https://www.rabbitmq.com/ttl.html#dead-letter-exchange](https://www.rabbitmq.com/ttl.html#dead-letter-exchange)
* Kafka consumer groups and offsets: [https://kafka.apache.org/documentation/#consumerconfigs](https://kafka.apache.org/documentation/#consumerconfigs)
## After the lab
You should now understand:
* How fan-out delivery duplicates a message for different delivery channels.
* Why duplicates happen and how redelivery semantics work.
* How dead-letter queues help capture and debug undeliverable notifications.
Use the broker UI or CLI to replay messages from the DLQ after fixing a worker bug to validate your recovery process.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.