Notifications That Survive

• A single event duplicated across multiple queues (fan-out delivery).
• Redelivery and duplicate notifications (at-least-once semantics and crash/retry behavior).
• Undeliverable or repeatedly failing notifications routed to a dead-letter queue (DLQ) for inspection or replay.

​

Prerequisites

• Running message broker (e.g., RabbitMQ, Kafka, or another AMQP/streaming system).
• Push and email worker processes that consume from separate queues.
• Dead-letter queue configured and bound according to your broker’s DLQ policy.
• Access to broker management UI or CLI to inspect queue depths and message metadata.

​

What you will learn

• How a single published event is duplicated across multiple queues so different workers can handle different delivery channels.
• Why duplicate deliveries occur (for example, at-least-once delivery, worker crashes, explicit negative acknowledgements).
• How a dead-letter queue collects messages that cannot be processed after configured attempts or are explicitly rejected.

​

High-level flow

1. A producer publishes an event to a fan-out exchange or routing mechanism.
2. The exchange routes a copy of the event to both the push queue and the email queue (fan-out).
3. Each worker consumes messages from its queue and attempts delivery:
   • On successful delivery the worker acknowledges (ACK) and the broker removes the message from that queue.
   • If a worker crashes or NACKs with requeue, the broker may redeliver the message, producing duplicates.
   • When a message exceeds the retry threshold or is explicitly rejected, the broker moves it to the dead-letter queue (DLQ).
4. The DLQ stores failed messages for inspection, debugging, or replay.

​

Key components and where to check

​

Example message payload

{
  "event_type": "user.signup",
  "user_id": "12345",
  "email": "user@example.com",
  "timestamp": "2026-10-07T12:34:56Z"
}

​

What to inspect during the lab

• Producer/publish log to confirm the event emission.
• Push worker logs and email worker logs to verify each receives and processes the event.
• Broker queue metadata:
  • Queue depths (ready vs unacked).
  • Message properties / headers (e.g., redelivered, delivery count, x-death).
• Dead-letter queue for messages that failed permanently and for diagnostic headers.

​

Troubleshooting tips

• Only one worker receives the event:
  • Verify the exchange type (fanout vs direct/topic).
  • Confirm both queues are bound to the exchange with the correct routing keys.
• Seeing duplicates:
  • Check worker stability and crash/restart logs.
  • Inspect the broker’s delivery semantics (at-least-once vs exactly-once capabilities).
  • Confirm the worker acknowledges messages only after successful processing.
• Messages in DLQ unexpectedly:
  • Review DLQ policy settings (max delivery attempts, TTL, rejection rules).
  • Check whether workers are rejecting messages or letting them timeout.
• No messages published:
  • Validate producer connection, authentication, and publish errors or retries.

​

Common broker troubleshooting commands and links

• RabbitMQ management UI: https://www.rabbitmq.com/management.html
• RabbitMQ dead-lettering: https://www.rabbitmq.com/ttl.html#dead-letter-exchange
• Kafka consumer groups and offsets: https://kafka.apache.org/documentation/#consumerconfigs

​

After the lab

• How fan-out delivery duplicates a message for different delivery channels.
• Why duplicates happen and how redelivery semantics work.
• How dead-letter queues help capture and debug undeliverable notifications.