Design a Home Feed For Instagram

​

Version 1 — Naive: compute feed at read time

SELECT *
FROM posts
WHERE author_id IN (`followed_user_ids`)
ORDER BY created_at DESC
LIMIT 50;

• Typical users follow ~200 people; power users may follow thousands. Even with indexes, the query must fetch, merge, and sort across many author timelines on every read.
• Users open the app many times per day, creating repeated heavy read queries that quickly overwhelm the database.

​

Version 2 — Fan-out on write (precompute feeds)

• Reads are cheap and fast: a single lookup of a per-user inbox.
• Eliminates runtime fan-in (merging many author timelines) on reads.

• Some accounts have tens or hundreds of millions of followers. A single celebrity post causes tens or hundreds of millions of inbox writes.
• The write pipeline and queueing system can be flooded, stalling the entire feed update process.

​

Version 3 — Hybrid push + pull

• Push (fan-out) for normal users (small–medium follower counts): when they post, push the post ID into followers’ inboxes.
• Pull for high-fanout authors (e.g., >100k followers — tuneable): store an author-centric feed for them (a small, recent list), and do not fan out their posts at write time.
• On feed load, merge:
  • The precomputed inbox (push results)
  • Recent posts fetched from author feeds for any large accounts the user follows
  • Re-rank and trim to the requested size

• Large fan-out storms are avoided because celebrity posts are pulled at read time.
• Most users still get the low-latency benefit of precomputed inboxes.

​

Version 4 — Move precomputed feeds out of the primary database into a cache

• Use the relational DB (Postgres, MySQL) as the canonical source of truth for posts and metadata.
• Store per-user inboxes in a fast in-memory store such as Redis. Typical data model:
  • Redis sorted set per user (score = timestamp) or a capped list
  • Trim inboxes to a reasonable cap (e.g., 800 entries)
• For celebrity/author feeds use a small author-centric cache (recent N posts) in Redis or a similar cache.

• Feed loads become single-digit milliseconds.
• Primary DB is protected from feed write/read churn.
• Celebrity posts are pulled from author caches, not the primary DB.

​

Final data-flow summary (recommended production architecture)

1. User uploads a photo
   • Write the post to the primary DB (Postgres) as the canonical record.
2. Fan-out service receives the write and decides:
   • If author is a “normal” user: push the post ID into each follower’s Redis inbox (sorted set), trimming to the inbox cap.
   • If author is a celebrity (above threshold): append the post to the author-feed cache and skip pushing to followers.
3. On feed load:
   • Read the user’s Redis inbox (fast).
   • For each followed celebrity above the threshold, fetch recent posts from the author-feed caches.
   • Merge inbox items and pulled author posts; re-rank by timestamp and/or relevance.
   • Fetch full post content from a post cache or from Postgres if needed, and perform privacy checks before rendering.

​

Tradeoffs at a glance

​

Implementation tips and operational knobs

• Celebrity threshold: e.g., 100_000 followers. Tune based on your traffic and write pipeline capacity.
• Redis inbox cap: e.g., 800 entries per user — most users do not scroll beyond a few hundred items.
• Fan-out batching and parallelism: batch writes to Redis and use parallel workers for pushes; use rate-limits for very large events.
• Cache TTLs and eviction: set sensible TTLs for author caches and post caches to balance freshness and cost.
• Observability: monitor tail latencies, queue backlogs, and Redis CPU/IO to detect fan-out storms.

​

Conclusion

​

Links and References

• Redis
• PostgreSQL
• Kubernetes Basics (for deployment/scale patterns)
• Articles on feed architectures and fan-out strategies for social applications