> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Notification System The Outbox
> Outbox pattern to ensure reliable notification delivery by making the database the source of truth, eliminating two-write races, and using an outbox processor plus idempotent workers
Before we continue, here's a concise review of the notification delivery flow we've designed and why we introduce the Outbox pattern.
## Original delivery flow (problem context)
1. A notification request arrives and we persist the notification record in the database with a unique notification ID and `status` set to `pending`.
2. We enqueue a small ticket (message) into the relevant queue for that notification's channel. We maintain one queue per channel — in our example we use four queues so a slow channel cannot block others.
3. A free worker picks up the task from the queue.
4. The worker checks a shared store (Redis) to see whether the idempotency key for this notification ID already exists. If it does, the worker drops the ticket to avoid duplicate delivery.
5. If the idempotency key is not present, the worker writes the key into Redis (with an expiry/TTL) and then calls the external provider to deliver the notification.
6. If the provider call fails, the worker retries using exponential backoff. If the notification keeps failing after several attempts, the worker moves the notification to a dead-letter queue for manual investigation by a support engineer.
7. If the provider call succeeds, the worker updates the notification `status` from `pending` to `sent` in the database.
8. Finally, the worker acknowledges the message to the queue so the queue removes the ticket permanently.
If a worker crashes at any point, the queue will redeliver the same ticket to another worker. The idempotency key stored in Redis ensures a duplicate ticket will be detected and dropped instead of being delivered twice.
The idempotency key must be set before calling the provider so concurrent workers don't result in duplicate deliveries. Give the key a reasonable TTL to allow retries and to expire after the delivery window has passed.
## The remaining reliability gap: the “two-writes” problem
There is one remaining reliability problem in the current flow: the notification server performs two separate writes.
* First write: persist the full notification record in the database.
* Second write: enqueue the ticket into the queue.
If the server successfully writes to the database but crashes before enqueuing the ticket, the notification is stored but never reaches the queue — and thus never gets delivered.
### Simple summary table
| Component | Responsibility | Example / Notes |
| - | -: | - |
| Database | Authoritative store of notifications and their `status` | `INSERT INTO notifications ... status = 'pending'` |
| Queue | Delivery tickets processed by workers | Channel-specific queue (4 queues in our example) |
| Worker | Consume tickets, enforce idempotency, call provider, update DB | Uses Redis idempotency key + TTL |
| Outbox processor | Scan DB for `pending` records and enqueue them | Ensures enqueue is derived from DB state |
## Solution: use the Outbox pattern
We fix the two-writes race by making the database the single source of truth and deriving enqueues from DB state.
Corrected approach:
1. When a notification request arrives, only store the notification in the database with `status = 'pending'`. Do not enqueue synchronously.
2. Run a small background job (an outbox processor) that periodically scans the database for notifications with `status = 'pending'` and enqueues them into the appropriate queue(s).
3. Workers pick up messages from the queues as before. If the background job inserts the same notification into the queue more than once, this is harmless because the worker-side idempotency checks will prevent duplicate sends.
This pattern — often called the Outbox pattern — ensures you perform a single durable write and have an asynchronous, reliable mechanism to publish messages to the queue. The database becomes the authoritative source of truth, eliminating the race condition when the server crashes between two writes.
## Implementation guidance and operational considerations
* Indexing & efficiency
* Index notifications by `status` and `created_at` so the outbox processor can efficiently find pending items.
* Batch enqueues where possible to reduce load on the queue and improve throughput.
* Outbox processor reliability
* Give the outbox scan job a retry/backoff policy for transient queue errors.
* Implement leader election (if running multiple outbox scanners) to avoid thundering herd, or partition scanning ranges.
* Idempotency and TTL
* Keep the idempotency TTL long enough to cover retries, possible redeliveries, and delays introduced by the outbox processor and the queue.
* Use the notification ID as the idempotency key to ensure uniqueness across workers.
* Error handling and observability
* Move repeatedly failing deliveries to a dead-letter queue for manual inspection.
* Emit metrics: enqueue rate, delivery successes/failures, retries, dead-letter counts, outbox backlog size.
* Alert on a growing outbox backlog or rising dead-letter queue volume.
Make sure the idempotency key is set in the shared store before calling the external provider. If you set it too late, concurrent delivery attempts can slip through and cause duplicate notifications.
## Example operational patterns
* Backoff policy for worker retries: exponential backoff with jitter and a max attempts threshold before sending the message to the dead-letter queue.
* Outbox scanning interval: tune based on traffic and required delivery latency (e.g., scan every few seconds for high-throughput systems, or every minute where slight delay is acceptable).
* Partitioning: for very large volumes, partition the outbox scan by notification channel or a hash range to allow concurrent, non-overlapping outbox workers.
## Links and references
* [Outbox pattern overview — Martin Fowler](https://martinfowler.com/)
* [Redis documentation — key expiry and best practices](https://redis.io/docs/)
* [Designing reliable message-driven systems — CQRS, Outbox, and idempotency patterns](https://en.wikipedia.org/wiki/CQRS)
With the Outbox pattern, the system becomes far more robust: the database is the source of truth, the outbox processor guarantees eventual enqueueing even if the initial server crashes, and worker-side idempotency prevents duplicate deliveries.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.