Designing a scalable resilient notification system using per-channel durable queues, fan-out of events, and robust worker behavior with retries, idempotency, DLQs, rate limiting, and observability
When an interviewer asks what kind of queue you should choose for a notification system, name these three essential properties:
Durable: the queue persists messages to disk (not only in RAM) so messages survive process or machine restarts.
Acknowledged delivery: a message is removed from the queue only when a worker confirms the work is done (acknowledgements).
Multi-worker support: the queue allows multiple workers to consume in parallel so you can scale out as load increases.
One property we deliberately relax is strict global ordering. If two unrelated notifications swap order (for example, Alan’s and John’s), it usually doesn’t matter. Relaxing strict ordering reduces coordination and makes scaling much easier.
Property
Why it matters
Example
Durable
Prevents message loss across restarts or crashes
Use a message broker with disk persistence (e.g., RabbitMQ, Kafka)
Acknowledged delivery
Ensures work is only removed on success
Worker ACKs message after provider confirms send
Multi-worker support
Enables horizontal scaling under load
Multiple worker processes/consumers for the same queue
This design separates the notification-accepting side (API) from the sending side (workers). In interviews, emphasize that this is asynchronous processing: the caller never waits for delivery; the API enqueues the notification and returns quickly. When there is a burst of notifications, durable queues absorb the load and worker pools can scale independently to catch up.
This pattern is asynchronous processing: the API enqueues work and returns immediately, while independent worker processes consume the queue and perform the actual sends.
Should all channels share a single queue (push, SMS, email, in‑app)? No — a shared queue risks head-of-line blocking. If one provider (for example, the email provider) is slow or rate-limited, workers will be tied up processing those slow messages and other channels will be effectively blocked.
To avoid head-of-line blocking, isolate channels by deploying one durable queue per channel. In the example below we have four queues (Push, SMS, Email, In‑app), each with its own worker pool. Each channel scales independently — a slow email provider affects only the email queue while push, SMS, and in‑app delivery continue unhindered.The in‑app channel is the simplest to implement because it does not depend on external providers. The notification already exists in our database; the in‑app worker only needs to mark the notification as delivered/seen. Typical in‑app flow:
When storing notifications, set a seen flag to false.
When the mobile app requests unseen notifications, the server queries rows where seen == false and returns them.
When the user views notifications, the app calls the server to mark them as seen (set seen = true).
Introduce the term fan-out: the notification service publishes a single event and creates a copy of that event in every channel queue. Copying one event into many queues is called fan-out.A concise summary of the design so far:
API (for example, POST /notify) records the notification in the database with a unique notification ID and immediately returns accepted.
The notification fans out into multiple durable queues (one per channel), so a slow provider can’t block other channels.
Workers consume each channel queue, look up contact details and preferences, and call providers to deliver notifications.
Worker responsibilities — what should a worker do when it receives a message?
Responsibility
Purpose / Notes
Validate and parse the message
Extract notification ID, channel, payload, and metadata
Lookup user contact details and preferences
Phone number, email, push token, and opt-in preferences
Use sensible timeouts to avoid long blocking calls
Acknowledge only after success
Remove the message from the queue on confirmed success
Retry on transient failures
Exponential backoff and limited retry attempts
Use a dead-letter queue (DLQ)
After max retries, move message to DLQ for manual inspection
Implement idempotency
Use notification ID to prevent duplicate deliveries on retries
Respect rate limits and throttling
Enforce per-provider or per-user concurrency limits
Log and emit metrics
Track success/failure rates, latency, retries, and traces
Concurrency limits
Avoid overloading downstream systems (per-provider/user limits)
Operational notes and best practices
Idempotency: Use the notification ID as the deduplication key with providers where possible. For example, store a provider-specific send ID in a durable store and skip duplicate sends.
Retries and DLQs: Configure exponential backoff and a maximum retry count. Send persistent failures to a DLQ for inspection and reprocessing.
Observability: Emit structured logs, metrics, and distributed traces for each send attempt so failures are diagnosable.
Rate limiting and throttling: Use token buckets or leaky buckets per provider to avoid hitting provider rate limits and cascading failures.
Concurrency controls: Limit concurrent sends per user or per provider to reduce risk of throttling or account-level blocking.
Putting it together: durable per-channel queues + fan-out + resilient worker semantics (timeouts, retries, idempotency, DLQ, metrics) yields a notification system that is resilient, scalable, and interview-ready.Links and references
