Skip to main content
Before we continue, here’s a concise review of the notification delivery flow we’ve designed and why we introduce the Outbox pattern.

​
Original delivery flow (problem context)

  1. A notification request arrives and we persist the notification record in the database with a unique notification ID and status set to pending.
  2. We enqueue a small ticket (message) into the relevant queue for that notification’s channel. We maintain one queue per channel — in our example we use four queues so a slow channel cannot block others.
  3. A free worker picks up the task from the queue.
  4. The worker checks a shared store (Redis) to see whether the idempotency key for this notification ID already exists. If it does, the worker drops the ticket to avoid duplicate delivery.
  5. If the idempotency key is not present, the worker writes the key into Redis (with an expiry/TTL) and then calls the external provider to deliver the notification.
  6. If the provider call fails, the worker retries using exponential backoff. If the notification keeps failing after several attempts, the worker moves the notification to a dead-letter queue for manual investigation by a support engineer.
  7. If the provider call succeeds, the worker updates the notification status from pending to sent in the database.
  8. Finally, the worker acknowledges the message to the queue so the queue removes the ticket permanently.
If a worker crashes at any point, the queue will redeliver the same ticket to another worker. The idempotency key stored in Redis ensures a duplicate ticket will be detected and dropped instead of being delivered twice.
The idempotency key must be set before calling the provider so concurrent workers don’t result in duplicate deliveries. Give the key a reasonable TTL to allow retries and to expire after the delivery window has passed.

​
The remaining reliability gap: the “two-writes” problem

There is one remaining reliability problem in the current flow: the notification server performs two separate writes.
  • First write: persist the full notification record in the database.
  • Second write: enqueue the ticket into the queue.
If the server successfully writes to the database but crashes before enqueuing the ticket, the notification is stored but never reaches the queue — and thus never gets delivered.

​
Simple summary table

​
Solution: use the Outbox pattern

We fix the two-writes race by making the database the single source of truth and deriving enqueues from DB state. Corrected approach:
  1. When a notification request arrives, only store the notification in the database with status = 'pending'. Do not enqueue synchronously.
  2. Run a small background job (an outbox processor) that periodically scans the database for notifications with status = 'pending' and enqueues them into the appropriate queue(s).
  3. Workers pick up messages from the queues as before. If the background job inserts the same notification into the queue more than once, this is harmless because the worker-side idempotency checks will prevent duplicate sends.
This pattern — often called the Outbox pattern — ensures you perform a single durable write and have an asynchronous, reliable mechanism to publish messages to the queue. The database becomes the authoritative source of truth, eliminating the race condition when the server crashes between two writes.

​
Implementation guidance and operational considerations

  • Indexing & efficiency
    • Index notifications by status and created_at so the outbox processor can efficiently find pending items.
    • Batch enqueues where possible to reduce load on the queue and improve throughput.
  • Outbox processor reliability
    • Give the outbox scan job a retry/backoff policy for transient queue errors.
    • Implement leader election (if running multiple outbox scanners) to avoid thundering herd, or partition scanning ranges.
  • Idempotency and TTL
    • Keep the idempotency TTL long enough to cover retries, possible redeliveries, and delays introduced by the outbox processor and the queue.
    • Use the notification ID as the idempotency key to ensure uniqueness across workers.
  • Error handling and observability
    • Move repeatedly failing deliveries to a dead-letter queue for manual inspection.
    • Emit metrics: enqueue rate, delivery successes/failures, retries, dead-letter counts, outbox backlog size.
    • Alert on a growing outbox backlog or rising dead-letter queue volume.
Make sure the idempotency key is set in the shared store before calling the external provider. If you set it too late, concurrent delivery attempts can slip through and cause duplicate notifications.

​
Example operational patterns

  • Backoff policy for worker retries: exponential backoff with jitter and a max attempts threshold before sending the message to the dead-letter queue.
  • Outbox scanning interval: tune based on traffic and required delivery latency (e.g., scan every few seconds for high-throughput systems, or every minute where slight delay is acceptable).
  • Partitioning: for very large volumes, partition the outbox scan by notification channel or a hash range to allow concurrent, non-overlapping outbox workers.
With the Outbox pattern, the system becomes far more robust: the database is the source of truth, the outbox processor guarantees eventual enqueueing even if the initial server crashes, and worker-side idempotency prevents duplicate deliveries.

Watch Video