The queue is the system’s heart
A durable queue is central to scalable notification systems. In interviews, call out concrete queue implementations such as Amazon SQS — a strong fit because it provides acknowledgement semantics and built-in dead-letter queues. RabbitMQ is another common option for systems that need more advanced routing. If an event causes a notification flood, queue lengths will grow and may not decrease until you increase processing capacity. A rising queue size is a clear signal to add more workers or increase throughput. Note that this design accepts duplicate tickets (see idempotency below).
Delivery semantics: at-least-once and idempotency
Queues often deliver the same message multiple times. This design intentionally uses at-least-once delivery: when the system is unsure whether a notification was delivered, it retries and may send the notification again. Sending a duplicate is preferable to losing a notification. To prevent duplicates from reaching users, perform an idempotency check just before calling the external provider. Use a fast store (e.g., Redis) to record a notification’s unique idempotency key so the provider is invoked only once per logical notification. With that guard, end users rarely see duplicates.At-least-once delivery is a practical choice for distributed systems: you can’t reliably guarantee exactly-once across distributed components, but you can make redelivery harmless using idempotency and deduplication.
Concise interview-ready notification flow
Use this short, structured answer when asked about notification delivery in an interview:- The notify API writes a notification record with a unique notification ID to the database and immediately returns Accepted.
- A background job enqueues a ticket so the notification can be delivered asynchronously.
- We keep one queue per channel (email, push, SMS, etc.) so a slow provider on one channel cannot block others.
- Workers pick tickets from the queues and consult
Redis(or a similar fast store) to check whether the notification has already been sent. - If it has not been sent, the worker evaluates the user’s preferences and proceeds only if delivery is allowed.
- The worker calls the external provider to send the notification.
- On failures we retry with exponential backoff; messages that still fail are routed to a dead-letter queue.
Components, responsibilities, and scaling signals
Use these metrics to decide when to scale horizontally (more workers) or vertically (bigger instances).
Alternatives: event streaming and centralized gateways
You can implement durable ingestion and buffering with event streaming platforms (e.g., Kafka) or classic queues. The conceptual pattern stays the same: durable ingestion → buffered processing → retries → dead-letter handling. Real companies extend this basic pattern:- LinkedIn initially had each team send notifications independently, producing duplicate emails. They centralized delivery in a gateway called Air Traffic Controller, which routes events from Kafka and centralizes delivery logic. This reduced emails by ~50% and complaints by more than 65%.
- Uber uses per-priority buckets for push notifications so urgent messages (e.g., trip updates) get higher priority than promotional offers. Priority queues ensure delivery order matches business urgency.
Best practices for production-ready notification systems
- Use durable ingestion (SQS / RabbitMQ / Kafka) to absorb bursts.
- Buffer and horizontally scale workers based on queue depth and latency.
- Enforce idempotency at the provider boundary (use
notification_idor an idempotency key stored inRedis). - Respect user preferences and apply delivery rules before provider calls.
- Implement exponential backoff and a dead-letter queue for permanent failures.
- Centralize routing and deduplication when multiple teams produce notifications (reduces user noise).
- Add priority buckets if different messages have different urgency.