> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Curveball Byte Notifications
> Concise interview-ready follow-ups for designing resilient notification systems covering priority queues, outage handling, idempotency limits, expiry, and multi-provider failover.
Towards the end of a system-design interview you’ll often face follow-up questions that probe how you'd harden your design for real production issues. This article covers three common, concrete follow-ups and shows concise, interview-ready answers you can use.
## Question 1 — A marketing campaign floods the queue with one million notifications. Alan’s login code is stuck behind that traffic. What do you change?
Problem: transactional messages (login codes, password resets) have strict latency needs (seconds), while marketing campaigns can tolerate much higher delays. A shared queue allows bulk marketing traffic to block critical transactional messages.
Solution:
* Segregate traffic by priority at enqueue time. Create separate queues per channel:
* `email.transactional`, `email.marketing`
* `sms.transactional`, `sms.marketing`
* `push.transactional`, `push.marketing`
* Run dedicated worker pools for each transactional queue. Workers for marketing queues must not be allowed to steal or borrow transactional workers.
* Optionally apply more granular priority buckets (e.g., high/medium/low) for complex workloads and SLA differentiation.
Benefits:
* Transactional messages stay low-latency even during large marketing campaigns.
* Easier to scale marketing workers independently without affecting transactional SLAs.
Separate queues and dedicated workers for transactional vs. marketing traffic preserve low latency for critical notifications.
Tip: large-scale platforms often use priority buckets or separate partitions to guarantee service levels — mention this if the interviewer asks about real-world precedents.
## Question 2 — The email provider is down for one hour. What does the system do during that hour?
Key behaviours to describe:
* Durable accumulation: email messages remain safely in the queue (persistent/durable storage) and are not lost.
* Backoff + circuit breaker: workers should stop hammering a failing provider after a configurable number of consecutive failures. Use an exponential backoff and open a circuit that prevents delivery attempts until a periodic probe detects recovery.
* Periodic probe for recovery: a single lightweight probe (for example, once per minute) can be used to test provider health and close the circuit when the provider is healthy again.
* Multi-provider failover: support a secondary (or active-active) provider and route traffic there if the primary is down.
* Respect expiry: short-lived notifications (e.g., OTPs) should carry an expiry timestamp; expired messages should be discarded rather than retried.
* Idempotency + audit: keep delivery logs and dedupe attempts where possible.
Message schema example (enqueue-time metadata):
```json theme={null}
{
"id": "notif-1234",
"type": "login_code",
"channel": "sms",
"priority": "transactional",
"expires_at": "2026-10-07T12:00:00Z",
"payload": { "code": "123456" }
}
```
Operational workflow during outage:
* Queue retains messages (no data loss).
* Workers hit the circuit breaker after N failures and switch to periodic probes.
* If a secondary provider exists, traffic is routed there automatically or via configurable failover rules.
* Expired messages are dropped at dequeue time.
Table: outage behavior vs. system action
| Event | System response |
| - | - |
| Provider returns transient errors | Backoff and retries with exponential delay |
| Provider fails repeatedly | Open circuit, stop attempts, run periodic probes |
| Provider recovers | Close circuit and resume normal delivery |
| Message expired while provider down | Drop message and optionally emit expiry metric/alert |
References:
* Circuit breaker pattern: [https://martinfowler.com/bliki/CircuitBreaker.html](https://martinfowler.com/bliki/CircuitBreaker.html)
* Exponential backoff: commonly used for transient network failures
## Question 3 — You say exactly-once delivery is impossible. Show exactly when the system could still send the notification twice and why an idempotency key might fail.
Explain the critical race that causes uncertainty:
Scenario (step-by-step):
1. A worker dequeues message `notif-1234` and checks Redis (or a sent-log) for whether `notif-1234` was delivered.
2. Redis indicates it has not been sent. The worker marks `notif-1234` as “in-progress” and calls the SMS provider.
3. The request to the provider times out before the worker receives a response.
At this point, three states are possible:
* Provider never received the request → safe to retry.
* Provider received and sent the message, but the response was lost → retry will cause duplicate delivery.
* Provider received and is still processing → retry may or may not duplicate delivery.
This uncertainty means the system cannot guarantee exactly-once end-to-end.
Mitigations (practical trade-offs):
* Idempotency keys: include the notification ID in the provider request. Many providers will deduplicate on server-side if the same idempotency key is presented.
* Delivery acknowledgements + durable audit log: only mark as delivered after a confirmed acknowledgement; but timeouts and provider semantics still leave ambiguity.
* User-level design: ensure the user experience tolerates rare duplicates (e.g., short-lived codes, clear messaging).
Worker pseudocode (simplified):
```text theme={null}
dequeue message -> msg
if msg.expires_at < now:
drop msg
elif redis.get("delivered:" + msg.id) == true:
drop msg
else:
redis.set("in-progress:" + msg.id, true)
response = provider.send(msg.payload, headers={"Idempotency-Key": msg.id})
if response.success:
redis.set("delivered:" + msg.id, true)
redis.delete("in-progress:" + msg.id)
else if response.timeout:
// cannot know; decide retry policy and rely on idempotency
```
Idempotency keys reduce provider-level duplicates, but network/carrier-level behavior can still cause duplicate deliveries. Exactly-once semantics cannot be fully guaranteed end-to-end.
Practical advice to discuss in interviews:
* Emphasize that idempotency greatly reduces duplicates but doesn't eliminate them across the full delivery chain (carrier/operator behaviors).
* Describe measurable SLAs and monitoring (duplicate-rate alerts, delivery latency histograms).
* Propose compensating UX or business rules for duplicates (e.g., ignore duplicate OTPs within a small time window).
## Summary — What to say in an interview
* Separate priorities: use dedicated queues and worker pools for transactional vs. marketing notifications to guarantee low latency for critical messages.
* Design for outages: durable queues, circuit breakers, periodic probes, and multi-provider failover keep the system resilient to provider outages.
* Respect expiry: attach expiry timestamps to short-lived notifications so workers discard stale messages.
* Mitigate duplicates: use idempotency keys and provider-side deduplication, but acknowledge end-to-end exactly-once delivery is infeasible; design the UX to tolerate rare duplicates.
Take a moment to internalize these follow-ups. Interviewers expect concise reasoning and trade-offs — show that you can identify practical mitigations, explain their limitations, and propose observability and recovery strategies.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.