> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Notification System One Queue Per Channel > Designing a scalable resilient notification system using per-channel durable queues, fan-out of events, and robust worker behavior with retries, idempotency, DLQs, rate limiting, and observability When an interviewer asks what kind of queue you should choose for a notification system, name these three essential properties: * Durable: the queue persists messages to disk (not only in RAM) so messages survive process or machine restarts. * Acknowledged delivery: a message is removed from the queue only when a worker confirms the work is done (acknowledgements). * Multi-worker support: the queue allows multiple workers to consume in parallel so you can scale out as load increases. One property we deliberately relax is strict global ordering. If two unrelated notifications swap order (for example, Alan's and John's), it usually doesn't matter. Relaxing strict ordering reduces coordination and makes scaling much easier. | Property | Why it matters | Example | | - | - | - | | Durable | Prevents message loss across restarts or crashes | Use a message broker with disk persistence (e.g., RabbitMQ, Kafka) | | Acknowledged delivery | Ensures work is only removed on success | Worker ACKs message after provider confirms send | | Multi-worker support | Enables horizontal scaling under load | Multiple worker processes/consumers for the same queue | A dark-themed presentation slide titled "What kind of queue?" with colorful diagrams showing three required properties—durable (on-disk vs RAM), acknowledged messages, and many workers for parallel processing. It also notes strict ordering isn't needed and includes example task boxes and worker icons. This design separates the notification-accepting side (API) from the sending side (workers). In interviews, emphasize that this is asynchronous processing: the caller never waits for delivery; the API enqueues the notification and returns quickly. When there is a burst of notifications, durable queues absorb the load and worker pools can scale independently to catch up. This pattern is asynchronous processing: the API enqueues work and returns immediately, while independent worker processes consume the queue and perform the actual sends. Should all channels share a single queue (push, SMS, email, in‑app)? No — a shared queue risks head-of-line blocking. If one provider (for example, the email provider) is slow or rate-limited, workers will be tied up processing those slow messages and other channels will be effectively blocked. A dark-themed diagram titled "One Slow Provider" showing a single shared queue feeding delivery workers for Push, SMS, Email and In‑App channels. The Email provider is highlighted in red as slow, causing tasks to get stuck and block the queue. To avoid head-of-line blocking, isolate channels by deploying one durable queue per channel. In the example below we have four queues (Push, SMS, Email, In‑app), each with its own worker pool. Each channel scales independently — a slow email provider affects only the email queue while push, SMS, and in‑app delivery continue unhindered. The in‑app channel is the simplest to implement because it does not depend on external providers. The notification already exists in our database; the in‑app worker only needs to mark the notification as delivered/seen. Typical in‑app flow: * When storing notifications, set a `seen` flag to `false`. * When the mobile app requests unseen notifications, the server queries rows where `seen == false` and returns them. * When the user views notifications, the app calls the server to mark them as `seen` (set `seen = true`). Introduce the term fan-out: the notification service publishes a single event and creates a copy of that event in every channel queue. Copying one event into many queues is called fan-out. A concise summary of the design so far: * API (for example, `POST /notify`) records the notification in the database with a unique notification ID and immediately returns accepted. * The notification fans out into multiple durable queues (one per channel), so a slow provider can't block other channels. * Workers consume each channel queue, look up contact details and preferences, and call providers to deliver notifications. A dark-themed high-level architecture diagram titled "The design so far" showing a caller POST /notify to a service that stores an ID and fans out jobs into parallel queues and processors which send notifications to Push, SMS, Email, and In‑app channels. Worker responsibilities — what should a worker do when it receives a message? | Responsibility | Purpose / Notes | | - | - | | Validate and parse the message | Extract `notification ID`, `channel`, payload, and metadata | | Lookup user contact details and preferences | Phone number, email, push token, and opt-in preferences | | Personalize and template the payload | Render provider-specific templates, localization, personalization | | Send to provider with timeout | Use sensible timeouts to avoid long blocking calls | | Acknowledge only after success | Remove the message from the queue on confirmed success | | Retry on transient failures | Exponential backoff and limited retry attempts | | Use a dead-letter queue (DLQ) | After max retries, move message to DLQ for manual inspection | | Implement idempotency | Use `notification ID` to prevent duplicate deliveries on retries | | Respect rate limits and throttling | Enforce per-provider or per-user concurrency limits | | Log and emit metrics | Track success/failure rates, latency, retries, and traces | | Concurrency limits | Avoid overloading downstream systems (per-provider/user limits) | Operational notes and best practices * Idempotency: Use the notification ID as the deduplication key with providers where possible. For example, store a provider-specific send ID in a durable store and skip duplicate sends. * Retries and DLQs: Configure exponential backoff and a maximum retry count. Send persistent failures to a DLQ for inspection and reprocessing. * Observability: Emit structured logs, metrics, and distributed traces for each send attempt so failures are diagnosable. * Rate limiting and throttling: Use token buckets or leaky buckets per provider to avoid hitting provider rate limits and cascading failures. * Concurrency controls: Limit concurrent sends per user or per provider to reduce risk of throttling or account-level blocking. Putting it together: durable per-channel queues + fan-out + resilient worker semantics (timeouts, retries, idempotency, DLQ, metrics) yields a notification system that is resilient, scalable, and interview-ready. Links and references * [Message queue durability patterns](https://en.wikipedia.org/wiki/Message_queue) * [Idempotency design patterns](https://martinfowler.com/articles/idempotency.html) * [Dead-letter queues and retry strategies](https://docs.microsoft.com/azure/service-bus-messaging/service-bus-dead-letter-queues) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.