News Feed Feeds Built in Advance

​

Core idea

• The source of truth remains the relational tables: users, posts, and follows.
• To guarantee sub-second feed loads, compute feeds before users open the app and store them in fast, in-memory caches (Redis).
• Each user has a small Redis list containing the most-recent post IDs for their feed. With 10M users, that becomes 10M small lists — so careful sizing and eviction are required.

​

Pre-computed feed pipeline (end-to-end)

1. A user (John) creates a post.
   • The create-post endpoint persists the post in the posts table and returns a post ID (example: 71).
2. A background worker performs fan-out-on-write.
   • The worker queries the follows table to determine John’s followers (for example, 800 followers; Alan is one of them).
   • The worker pushes the post ID (71) to each follower’s feed-index list in Redis.
   • Use atomic push-and-trim to keep lists bounded (e.g., LPUSH + LTRIM), batching and parallelizing writes for very large fan-outs.

LPUSH feed:alan 71
LTRIM feed:alan 0 199         # keep the most recent 200 items

for batch in follower_batches:
  pipeline = redis.pipeline()
  for follower in batch:
    pipeline.lpush("feed:" + follower_id, post_id)
    pipeline.ltrim("feed:" + follower_id, 0, MAX_FEED_LENGTH-1)
  pipeline.execute()

​

What the Redis feed-index stores

• Each per-user Redis list contains only numeric post IDs (e.g., 71, 68, 64), ordered newest-first.
• Lists are bounded to a configured maximum length (most-recent N IDs) to cap memory usage.
• Storing only IDs keeps the index compact and extremely fast to read.

​

Why we add a second cache (the post cache)

1. Read Alan’s list of post IDs from the feed-index cache.
2. Bulk-fetch corresponding post content from the post cache.
3. For any cache misses, fetch content from the posts table and populate the post cache to reduce future DB hits.

• Reduces duplicate database reads for shared posts.
• Enables batch retrieval of post content, minimizing network round-trips.

​

Two caches — roles and responsibilities

1. Get ID list from feed-index.
2. Multi-get content from post cache.
3. Fetch missing posts from DB and populate post cache.

​

Operational considerations

• Memory footprint and sharding
  • 10M Redis lists requires bounding lists and planning for sharding/partitioning. Size instances and clusters based on average list length and desired retention window.
• Post cache eviction
  • Use TTLs or LRU for the post cache so it retains recent/popular posts while freeing memory for new content.
• Heavy-fanout writers (celeb accounts)
  • Posts from users with millions of followers spike the fan-out workload and write amplification. Consider hybrid strategies:
    • Hybrid fan-out: fan-out-on-write for most users, fan-out-on-read or per-followers paging for celebrities.
    • Tiered handling: route celebrity fan-outs to dedicated worker pools and partition the write operations.
• Batching and network efficiency
  • Fetch post content in batches (multi-get) from the post cache to minimize round-trips.
  • When writing fan-out updates, pipeline/batch Redis commands.
• Consistency for edits/deletes
  • Edits: update post cache entries and rely on the DB for authoritative content; eventual consistency is often acceptable for feeds.
  • Deletes: invalidate post cache and optionally remove IDs from feed-index lists, or let the feed read-time handle missing content gracefully.
• Monitoring and metrics
  • Track follower distribution, cache hit ratios (post cache and feed-index reads), write latency for fan-outs, and Redis memory usage.

​

Handling extremes: celebrities and very large fans

• For accounts with extremely large follower counts:
  • Consider publishing the post to a separate “celebrity” store and dynamically merging that content at read time (fan-out-on-read for that subset).
  • Implement rate-limited, parallelized worker pools to spread the fan-out write load across many workers and Redis shards.
  • Use partial fan-out (seed a subset of followers proactively; serve remaining followers via on-read aggregation).
• Use backpressure and queueing (e.g., Kafka, SQS) to smooth spikes.

​

Consistency and maintenance patterns

• Update or edit flows:
  • Update the posts table first, then update/invalidate the post cache.
  • Optionally update feed-index lists if you need post ID ordering changes (usually not required).
• Delete flows:
  • Delete from DB, invalidate post cache. Optionally remove IDs from feed-index lists or let missing content render as “deleted” during read.
• Rebuild strategies:
  • If you must rebuild a feed-index for a user, run a background job that re-computes recent IDs from the follows set and recent posts, writing into Redis in a controlled way.

​

Implementation notes and resources

• Use Redis lists (or sorted sets if you need scores/timestamps) for feed-indexes. See Redis docs: https://redis.io/
• Use a high-performance in-memory key-value store for the post cache (Redis, Memcached, or an in-process cache with distributed invalidation).
• Consider using a durable queue (Kafka, RabbitMQ, SQS) to buffer fan-out tasks and smooth spikes.
• Monitor metrics: Redis memory usage, post cache hit rate, fan-out write latency, and request tail latency.

​

References

• Redis documentation: https://redis.io/
• Cache-invalidation patterns and strategies: https://martinfowler.com/articles/cache-invalidation.html
• Scalable feed architectures (overview articles and papers)