News Feed The Background Job

This article explains a robust, interview-ready implementation of the background job that performs fan-out for a news feed. The approach centers on the transactional outbox pattern: record the new post and the delivery intent atomically, then reliably deliver that intent to a queue so worker processes can populate followers’ feeds. When John creates a new post from his phone, the client calls the server’s create-post endpoint. The server opens a database transaction and performs two writes inside that single transaction:

Insert the new post into the posts table.
Insert a small record into the outbox table that records the delivery intent (for example: post ID 71, status pending, and optional payload).

Because both inserts occur inside the same transaction, they are atomic: either both succeed or neither does. Example SQL inside the transaction:

BEGIN;

INSERT INTO posts (id, author_id, body, created_at)
VALUES (71, 123, 'Good morning', now());

INSERT INTO outbox (post_id, status, payload, created_at)
VALUES (71, 'pending', '{"post_id":71, "author_id":123}', now());

COMMIT;

A separate dispatcher/poller (background process) watches the outbox table and finds rows with status = 'pending'. Typical dispatchers either poll every few seconds or use a database event/notify mechanism. A robust poller first claims rows atomically to avoid duplicate claims from multiple pollers (for example, update pending → processing with a lease or use FOR UPDATE SKIP LOCKED). Poller responsibilities (high level):

Claim pending outbox rows safely (avoid races).
Send a message/ticket to the external message queue describing the work (e.g., { "post_id": 71, "author_id": 123 }).
Mark the outbox row done (or remove/archive it) only after the queue acknowledges delivery, or rely on visibility/lease semantics to enable retries.

Example claiming + enqueue pseudocode:

-- Claim a batch of pending rows (Postgres example)
WITH candidates AS (
  SELECT id FROM outbox
  WHERE status = 'pending'
  ORDER BY created_at
  LIMIT 100
  FOR UPDATE SKIP LOCKED
)
UPDATE outbox
SET status = 'processing', lease_expires = now() + interval '30 seconds'
FROM candidates
WHERE outbox.id = candidates.id
RETURNING outbox.id, outbox.payload;

Then the poller enqueues returned payloads to the message queue and, once the queue confirms (ACK), updates the rows to done.

A dark-themed technical diagram titled "The Outbox Pattern" showing an app architecture with users (John, Alan), an app server, database tables, Redis feed/post caches, and a background worker/queue. Arrows illustrate data flow between components and an example post entry ("71 · good morning").

Once a ticket is in the queue, worker nodes (consumers) pick it up and perform the fan-out: Worker processing steps:

Read the ticket (e.g., post ID 71, author John).
Query the follows table to list John’s followers (for example, 800 followers).
Push the new post into each follower’s feed/store (Redis lists, sorted sets, or a feed table).
Acknowledge the ticket (so the queue marks the job complete).

Typical worker pseudocode:

ticket = queue.receive()
post_id = ticket['post_id']
author_id = ticket['author_id']

followers = db.query("SELECT follower_id FROM follows WHERE followee_id = %s", author_id)

for f in followers:
    # idempotent insert or upsert to avoid duplicates
    redis.zadd(f"feed:{f}", {post_id: post_timestamp})

queue.ack(ticket)

Failure modes and deduplication One important failure mode: a worker may push the post into followers’ feeds and then crash before acknowledging the ticket. Another worker may then receive the same ticket and reapply the same post, creating duplicates (e.g., Alan’s feed containing the post twice). Common strategies to avoid or mitigate duplicates:

Failure mode	Mitigation
Worker re-applies the same post after a crash (duplicate feed entries)	Make consumers idempotent: check existence before insert, or use an atomic upsert/replace.
Storage allows duplicate entries (e.g., Redis list)	Use data structures that inherently deduplicate, such as `SET`/`ZSET` keyed by `post_id`, or add a unique constraint on `(follower_id, post_id)` in a relational feed table.
Outbox message lost before enqueue acknowledgement	Ensure poller only marks outbox rows `done` after receiving an ACK from the external queue; or use a transactional dispatcher / DB-backed queue to reduce the windows for loss.

A robust design assumes at-least-once delivery from the queue and therefore makes consumers and storage idempotent or de-duplicating. For example, enforce a unique constraint:

ALTER TABLE feed_entries ADD CONSTRAINT unique_feed_post UNIQUE (follower_id, post_id);

Or use a Redis ZADD with the NX or check-before-insert pattern. Why the outbox pattern?

It gives you a transactional guarantee between your domain write (the new post) and the intent to deliver that event.
It decouples the critical user-facing transaction from the (potentially slow) fan-out work.
When combined with idempotent consumers and careful dispatch semantics, it yields a reliable, production-ready system for fan-out at scale.

Using the outbox pattern gives you a transactional guarantee between your domain write and the delivery intent. Pair it with idempotent consumers or storage-level deduplication to make overall delivery reliable and resilient to crashes.

Practical notes and trade-offs

At-large scale, pushing a post synchronously to thousands of followers can be expensive; outbox + queue + workers lets you smooth that work asynchronously.
At-least-once delivery is the practical default for many queues. Building idempotency into consumers is usually simpler and faster than trying to achieve exactly-once semantics.
In some products, occasional duplicates that are corrected upon refresh are acceptable. In consumer-facing production systems you should still prefer idempotent processing or deduplication for a clean UX.

Links and references

Transactional Outbox Pattern — Martin Fowler
Postgres: FOR UPDATE SKIP LOCKED
Designing Data-Intensive Applications (book, for deeper patterns in distributed systems)

The Interview, Replayed

Warm-Ups: The Read-Heavy Classics

Fan-Out: One Event, Many Readers

News Feed The Background Job

Watch Video