> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# News Feed The Background Job
> Explains using the transactional outbox, dispatcher, and idempotent workers to reliably fan out posts to followers' feeds and prevent duplicates.
This article explains a robust, interview-ready implementation of the background job that performs fan-out for a news feed. The approach centers on the transactional outbox pattern: record the new post and the delivery intent atomically, then reliably deliver that intent to a queue so worker processes can populate followers' feeds.
When John creates a new post from his phone, the client calls the server's create-post endpoint. The server opens a database transaction and performs two writes inside that single transaction:
1. Insert the new post into the `posts` table.
2. Insert a small record into the `outbox` table that records the delivery intent (for example: post ID `71`, status `pending`, and optional payload).
Because both inserts occur inside the same transaction, they are atomic: either both succeed or neither does.
Example SQL inside the transaction:
```sql theme={null}
BEGIN;
INSERT INTO posts (id, author_id, body, created_at)
VALUES (71, 123, 'Good morning', now());
INSERT INTO outbox (post_id, status, payload, created_at)
VALUES (71, 'pending', '{"post_id":71, "author_id":123}', now());
COMMIT;
```
A separate dispatcher/poller (background process) watches the `outbox` table and finds rows with `status = 'pending'`. Typical dispatchers either poll every few seconds or use a database event/notify mechanism. A robust poller first claims rows atomically to avoid duplicate claims from multiple pollers (for example, update `pending` → `processing` with a lease or use `FOR UPDATE SKIP LOCKED`).
Poller responsibilities (high level):
* Claim pending outbox rows safely (avoid races).
* Send a message/ticket to the external message queue describing the work (e.g., `{ "post_id": 71, "author_id": 123 }`).
* Mark the outbox row `done` (or remove/archive it) only after the queue acknowledges delivery, or rely on visibility/lease semantics to enable retries.
Example claiming + enqueue pseudocode:
```sql theme={null}
-- Claim a batch of pending rows (Postgres example)
WITH candidates AS (
SELECT id FROM outbox
WHERE status = 'pending'
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED
)
UPDATE outbox
SET status = 'processing', lease_expires = now() + interval '30 seconds'
FROM candidates
WHERE outbox.id = candidates.id
RETURNING outbox.id, outbox.payload;
```
Then the poller enqueues returned payloads to the message queue and, once the queue confirms (ACK), updates the rows to `done`.
Once a ticket is in the queue, worker nodes (consumers) pick it up and perform the fan-out:
Worker processing steps:
1. Read the ticket (e.g., post ID `71`, author `John`).
2. Query the `follows` table to list John’s followers (for example, 800 followers).
3. Push the new post into each follower's feed/store (Redis lists, sorted sets, or a feed table).
4. Acknowledge the ticket (so the queue marks the job complete).
Typical worker pseudocode:
```python theme={null}
ticket = queue.receive()
post_id = ticket['post_id']
author_id = ticket['author_id']
followers = db.query("SELECT follower_id FROM follows WHERE followee_id = %s", author_id)
for f in followers:
# idempotent insert or upsert to avoid duplicates
redis.zadd(f"feed:{f}", {post_id: post_timestamp})
queue.ack(ticket)
```
Failure modes and deduplication
One important failure mode: a worker may push the post into followers' feeds and then crash before acknowledging the ticket. Another worker may then receive the same ticket and reapply the same post, creating duplicates (e.g., Alan's feed containing the post twice).
Common strategies to avoid or mitigate duplicates:
| Failure mode | Mitigation |
| - | - |
| Worker re-applies the same post after a crash (duplicate feed entries) | Make consumers idempotent: check existence before insert, or use an atomic upsert/replace. |
| Storage allows duplicate entries (e.g., Redis list) | Use data structures that inherently deduplicate, such as `SET`/`ZSET` keyed by `post_id`, or add a unique constraint on `(follower_id, post_id)` in a relational feed table. |
| Outbox message lost before enqueue acknowledgement | Ensure poller only marks outbox rows `done` after receiving an ACK from the external queue; or use a transactional dispatcher / DB-backed queue to reduce the windows for loss. |
A robust design assumes at-least-once delivery from the queue and therefore makes consumers and storage idempotent or de-duplicating. For example, enforce a unique constraint:
```sql theme={null}
ALTER TABLE feed_entries ADD CONSTRAINT unique_feed_post UNIQUE (follower_id, post_id);
```
Or use a Redis `ZADD` with the `NX` or check-before-insert pattern.
Why the outbox pattern?
* It gives you a transactional guarantee between your domain write (the new post) and the intent to deliver that event.
* It decouples the critical user-facing transaction from the (potentially slow) fan-out work.
* When combined with idempotent consumers and careful dispatch semantics, it yields a reliable, production-ready system for fan-out at scale.
Using the outbox pattern gives you a transactional guarantee between your domain write and the delivery intent. Pair it with idempotent consumers or storage-level deduplication to make overall delivery reliable and resilient to crashes.
Practical notes and trade-offs
* At-large scale, pushing a post synchronously to thousands of followers can be expensive; outbox + queue + workers lets you smooth that work asynchronously.
* At-least-once delivery is the practical default for many queues. Building idempotency into consumers is usually simpler and faster than trying to achieve exactly-once semantics.
* In some products, occasional duplicates that are corrected upon refresh are acceptable. In consumer-facing production systems you should still prefer idempotent processing or deduplication for a clean UX.
Links and references
* [Transactional Outbox Pattern — Martin Fowler](https://martinfowler.com/articles/patterns-of-distributed-systems/outbox.html)
* [Postgres: FOR UPDATE SKIP LOCKED](https://www.postgresql.org/docs/current/static/explicit-locking.html)
* [Designing Data-Intensive Applications](https://dataintensive.net/) (book, for deeper patterns in distributed systems)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.