Feed Fan Out

​

How fan-out-on-write works (high level)

1. A user creates a post. The post is written to the canonical posts table/stream.
2. The publish event is appended to a durable message stream or queue (Kafka, Pulsar, Kinesis, Redis Streams, etc.).
3. A fan-out worker consumes the stream and computes the follower set for the author.
4. The worker writes a per-follower feed entry (often a small summary or pointer to the canonical post) into each follower’s personal feed store.
5. Followers read their personal feed directly; reads are fast because feeds were pre-populated.

​

Example worker (pseudocode)

# fanout_worker.py
BATCH_SIZE = 100
MAX_RETRIES = 5

def fanout(post):
    # fetch followers for the author (list of follower IDs)
    followers = get_followers(post.author_id)

    # write entries in batches to reduce RPC overhead and improve throughput
    for batch in chunked(followers, BATCH_SIZE):
        entries = []
        for follower_id in batch:
            # stable idempotency key: combine post id and follower id
            entry_id = f"{post.id}:{follower_id}"

            # keep the per-follower payload small: pointer + summary + metadata
            entry = {
                "entry_id": entry_id,
                "post_id": post.id,
                "author_id": post.author_id,
                "created_at": post.created_at,
                "content_summary": post.summary
            }
            entries.append((follower_id, entry))

        # write the batch to the per-follower feed store (atomic/batched API if available)
        write_batch_to_feed_store(entries)

def worker_loop():
    # consume the canonical posts stream
    for post in consume_stream("posts"):
        try:
            fanout(post)
        except TransientError:
            retry_with_backoff(post, MAX_RETRIES)
        except FatalError:
            # log, alert, and move on; avoid blocking the stream for long periods
            log_and_alert(post)

• Use entry_id as an idempotency key so retries do not produce duplicates.
• Keep per-follower entries small (summary or pointer) to limit write amplification and storage costs.
• Batch follower writes to reduce RPC overhead and increase throughput.
• Persist a durable marker (or offset) of processed events so the consumer can resume safely.
• For very large follower lists, shard the fan-out work across multiple workers by author, by post, or by follower ranges.

​

Storage options for per-follower feeds

​

Important operational considerations

• Write amplification: one post may be written once per follower. Expect higher IOPS and storage usage.
• Hot authors: accounts with millions of followers create hot shards and cause large write spikes.
• Ordering: preserve timestamps; use monotonic counters or server-side timestamps if strict ordering is required.
• Consistency: fan-out is typically eventually consistent — followers may see a brief delay while writes propagate.
• Idempotency: use stable entry keys to make retries safe.
• Backpressure: handle spikes using batching, sharding, worker autoscaling, and circuit breakers.
• TTL and compaction: enforce retention (TTL) or compaction to bound per-user storage.
• Privacy and deletes: define a strategy for deletes and privacy changes (cascade deletes, tombstones, or access checks at read time).
• Monitoring: track consumer lag, queue depth, fan-out latency, write error rates, and per-shard throughput.

​

Scaling patterns and hybrids

• Fan-out on write (proactive): precompute and store each follower’s feed. Best: read latency. Trade-off: highest write cost and storage duplication.
• Fan-out on read (lazy): store canonical posts and follower lists; assemble feed at read time. Best: reduced write amplification. Trade-off: higher read latency and more CPU at read time.
• Hybrid approaches:
  • Fan-out to active followers (heavy readers) and fan-in on read for inactive followers.
  • Treat super-followed authors specially: store their posts in a shared high-throughput timeline that is merged at read time.
  • Use tiered storage: keep recent items in Redis for low latency; older items in cold storage (wide-column DB).

​

Summary

​

Links and references

• Kafka: https://kafka.apache.org/
• Pulsar: https://pulsar.apache.org/
• AWS Kinesis: https://aws.amazon.com/kinesis/
• Redis Streams: https://redis.io/topics/streams-intro
• Cassandra Data Modeling: https://cassandra.apache.org/doc/latest/
• DynamoDB Best Practices: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html

​

Example checklist before production rollout

• Implement idempotent writes (entry_id per follower).
• Add batching and worker sharding.
• Define TTL/retention policy and compaction strategy.
• Add monitoring for consumer lag, write failures, and hot keys.
• Design special handling for high-follower-count accounts.
• Test recovery and resume semantics for the consumer stream (offset management).