Skip to main content
We ran into a subtle but common failure mode when using a queue to drive notification delivery. Example scenario:
  • A worker pulls Alan’s payment-notification ticket from the queue.
  • The worker reads Alan’s details from the database and calls the SMS provider.
  • The SMS is delivered to Alan’s phone.
  • Before the worker can acknowledge the ticket to the queue, it crashes.
From the queue’s perspective the ticket was picked up and never acknowledged, so the queue re-delivers the same ticket to another worker. That second worker repeats the flow and the SMS is sent a second time — Alan receives duplicate notifications. You can see the issue: a crash between the provider call and the queue acknowledgement can cause duplicate external deliveries. Relying on “careful” worker code is insufficient because a crash can always occur at the worst moment.
A dark-themed system diagram titled "The Duplicate Trap" showing a notification server writing to a queue and worker(s) sending messages to an SMS provider, resulting in a phone receiving the same payment SMS twice. The flow highlights the queue, worker, and duplicate delivery issue.
Pause and think: how can we prevent one notification from being sent two times? The right approach is to accept that the queue can redeliver messages, and make duplicate processing harmless. Introduce a shared mechanism that detects and drops duplicates before calling the external provider — in other words, make processing idempotent. High-level solution
  • Give every notification a stable notification ID (stored in the DB).
  • Use a fast shared store (e.g., Redis) to hold a temporary “claim” for that notification ID before calling the provider.
    • If a worker finds the claim already present, it drops the ticket (another worker is handling it).
    • If no claim exists, the worker creates one (claiming ownership), then calls the provider.
  • Create the claim using an atomic “set-if-not-exists” with an expiry (TTL). The claim value should be a unique owner token so you can safely release it only if you own it.
  • If the worker crashes after creating the claim but before calling the provider, the claim expires and another worker can retry later.
  • If the provider call fails with an uncertain outcome, the worker removes its claim only if the owner token still matches (compare-and-delete). If the send succeeds, the worker updates the DB status to sent.
This small temporary claim is called an idempotency key. In this design the idempotency key is the notification ID; the stored value is the unique owner token. Step-by-step sequence (recommended)
  1. Worker pulls job from queue.
  2. Read notification payload and notification ID from DB.
  3. Generate a unique owner token (for example, a UUID).
  4. Atomically attempt to create the idempotency claim with TTL: SET key value NX EX <seconds>.
    • Key = notification ID
    • Value = owner token
  5. If SET NX EX succeeds:
    • Call the provider.
    • If call returns success:
      • Update DB notification status to sent.
      • Optionally refresh TTL while doing DB work to prevent expiry during long updates.
      • Remove the claim (best-effort) only if owner token still matches.
    • If call fails with a clear non-retryable error:
      • Remove claim only if owner token matches, then mark DB appropriately.
    • If call fails with uncertain outcome (timeout/no definitive response):
      • Remove claim only if owner token matches so another worker can retry.
  6. If SET NX EX fails (claim already present):
    • Drop the job; some other worker is processing it.
    • Optionally back off and observe DB status to verify.
Important operational details
  • Use an atomic set-if-not-exists with expiry (e.g., Redis SET key value NX EX seconds) when creating the claim.
  • Store a unique owner token as the value so you can perform a safe compare-and-delete later:
    • Example safe delete Lua script for Redis:
  • If the provider call succeeds, mark the notification as sent in the database.
  • If the provider call fails with no clear outcome, remove the claim only if your owner token matches — this avoids deleting a new claim that another worker set after your claim expired.
  • Choose the TTL long enough to cover the provider call and the DB update, or refresh the TTL while working. A TTL that is too short can allow duplicates if it expires before the send (or DB update) completes.
Idempotency means that no matter how many times you perform an operation, the observable result is the same. Here, regardless of how many times a ticket is processed, the notification is sent at most once.
Failure modes and trade-offs
  • Duplicate send (original problem) → solved by idempotency claims.
  • Lost notification (new risk) → can happen if a worker writes the claim and crashes before the provider call completes; the TTL prevents permanent loss by allowing retries after expiry. Pick TTL carefully.
  • Claim deletion race → solved by storing owner token and using compare-and-delete.
  • Increased latency in worst case → a claim expiry can delay delivery by up to TTL; this is generally preferable to performing duplicate external sends.
Quick reference table Implementation hints and tools
  • Redis is a good fit for the idempotency store because it supports SET NX EX and Lua scripting for atomic compare-and-delete.
  • Ensure worker clocks and TTLs are tuned to cover network and provider variability.
  • Consider refreshing the TTL periodically while the provider call + DB update are still in progress to avoid accidental expiry.
  • When generating owner tokens, use strong randomness (UUID v4 or similar).
Make sure the TTL covers the expected end-to-end time (provider call + DB update). If TTL is too short, the claim may expire and allow a duplicate send; if TTL is too long, recovery after a true failure can be delayed.
References With this idempotency-claim pattern the worst-case outcome becomes a slight delay (claim expiry + retry) rather than an accidental duplicate external delivery — a much safer and more user-friendly trade-off.

Watch Video