> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Feed Fan Out
> Explains building a low latency news feed using fan-out-on-write, including worker design, storage tradeoffs, operational best practices, and hybrid scaling strategies.
We’ll build a real news feed using the fan-out-on-write pattern. With fan-out-on-write, when an author publishes a post, a worker proactively copies that post into each follower’s personal feed storage. The benefit: follower reads are pre-populated and return instantly. The trade-off: increased write volume and duplicated storage.
This article describes the end-to-end flow, provides a simple worker implementation (with batching and idempotency), compares storage choices, and lists operational best practices and scaling patterns.
## How fan-out-on-write works (high level)
1. A user creates a post. The post is written to the canonical posts table/stream.
2. The publish event is appended to a durable message stream or queue (Kafka, Pulsar, Kinesis, Redis Streams, etc.).
3. A fan-out worker consumes the stream and computes the follower set for the author.
4. The worker writes a per-follower feed entry (often a small summary or pointer to the canonical post) into each follower’s personal feed store.
5. Followers read their personal feed directly; reads are fast because feeds were pre-populated.
This is the simplest end-to-end flow for achieving very low read latency for user feeds.
## Example worker (pseudocode)
The following Python-like pseudocode demonstrates a fan-out worker loop. It emphasizes batching, idempotency, and retry behavior.
```python theme={null}
# fanout_worker.py
BATCH_SIZE = 100
MAX_RETRIES = 5
def fanout(post):
# fetch followers for the author (list of follower IDs)
followers = get_followers(post.author_id)
# write entries in batches to reduce RPC overhead and improve throughput
for batch in chunked(followers, BATCH_SIZE):
entries = []
for follower_id in batch:
# stable idempotency key: combine post id and follower id
entry_id = f"{post.id}:{follower_id}"
# keep the per-follower payload small: pointer + summary + metadata
entry = {
"entry_id": entry_id,
"post_id": post.id,
"author_id": post.author_id,
"created_at": post.created_at,
"content_summary": post.summary
}
entries.append((follower_id, entry))
# write the batch to the per-follower feed store (atomic/batched API if available)
write_batch_to_feed_store(entries)
def worker_loop():
# consume the canonical posts stream
for post in consume_stream("posts"):
try:
fanout(post)
except TransientError:
retry_with_backoff(post, MAX_RETRIES)
except FatalError:
# log, alert, and move on; avoid blocking the stream for long periods
log_and_alert(post)
```
Notes about the pseudocode:
* Use `entry_id` as an idempotency key so retries do not produce duplicates.
* Keep per-follower entries small (summary or pointer) to limit write amplification and storage costs.
* Batch follower writes to reduce RPC overhead and increase throughput.
* Persist a durable marker (or offset) of processed events so the consumer can resume safely.
* For very large follower lists, shard the fan-out work across multiple workers by author, by post, or by follower ranges.
## Storage options for per-follower feeds
Choose the per-follower feed store based on your read-latency requirements, write throughput, storage cost, and operational familiarity.
| Storage type | Pros | Cons | Example / Notes |
| - | - | - | - |
| Redis lists / streams | Extremely low read latency; simple APIs (`LPUSH`, `LRANGE`) | Memory-backed — expensive at scale; requires eviction/TTL strategy | Good for hot-user caches and small-to-medium scale Redis clusters |
| Wide-column stores (Cassandra, Bigtable) | Scales to massive datasets; predictable disk-backed cost; per-user partitioning | Requires careful data modeling and compaction; write amplification affects I/O | Store each feed as a partition/row keyed by `user_id` and sorted by timestamp |
| DynamoDB / key-value stores | Managed, single-digit-millisecond reads with proper keys; good for variable scale | Cost per write & storage; requires correct partition key design to avoid hot partitions | Use `partition_key = user_id`, `sort_key = created_at` or `entry_id` |
| Relational DB per-user tables | Familiar transactional semantics; easy for small-scale systems | Hard to scale for large user bases; schema and indexing management | Suitable only for prototypes or low-scale systems |
## Important operational considerations
* Write amplification: one post may be written once per follower. Expect higher IOPS and storage usage.
* Hot authors: accounts with millions of followers create hot shards and cause large write spikes.
* Ordering: preserve timestamps; use monotonic counters or server-side timestamps if strict ordering is required.
* Consistency: fan-out is typically eventually consistent — followers may see a brief delay while writes propagate.
* Idempotency: use stable entry keys to make retries safe.
* Backpressure: handle spikes using batching, sharding, worker autoscaling, and circuit breakers.
* TTL and compaction: enforce retention (TTL) or compaction to bound per-user storage.
* Privacy and deletes: define a strategy for deletes and privacy changes (cascade deletes, tombstones, or access checks at read time).
* Monitoring: track consumer lag, queue depth, fan-out latency, write error rates, and per-shard throughput.
Accounts with extremely large follower counts (celebrities) are a major operational risk: they can cause sustained high write volumes and create hot partitions. Plan special handling (hybrid fan-out, rate-limiting, separate shards, or read-time fetch for followers) to protect the rest of the system.
## Scaling patterns and hybrids
* Fan-out on write (proactive): precompute and store each follower’s feed. Best: read latency. Trade-off: highest write cost and storage duplication.
* Fan-out on read (lazy): store canonical posts and follower lists; assemble feed at read time. Best: reduced write amplification. Trade-off: higher read latency and more CPU at read time.
* Hybrid approaches:
* Fan-out to active followers (heavy readers) and fan-in on read for inactive followers.
* Treat super-followed authors specially: store their posts in a shared high-throughput timeline that is merged at read time.
* Use tiered storage: keep recent items in Redis for low latency; older items in cold storage (wide-column DB).
## Summary
Fan-out-on-write pre-populates followers’ feeds at write time to deliver instant reads. It is ideal when read latency is the priority and extra write/storage cost is acceptable. To operate this pattern at scale, pay attention to batching, idempotency, hot-key handling, retention policies, and observability.
Fan-out-on-write delivers excellent read performance but increases write amplification and storage. Use hybrid strategies and special-case hot authors to balance cost and latency.
## Links and references
* Kafka: [https://kafka.apache.org/](https://kafka.apache.org/)
* Pulsar: [https://pulsar.apache.org/](https://pulsar.apache.org/)
* AWS Kinesis: [https://aws.amazon.com/kinesis/](https://aws.amazon.com/kinesis/)
* Redis Streams: [https://redis.io/topics/streams-intro](https://redis.io/topics/streams-intro)
* Cassandra Data Modeling: [https://cassandra.apache.org/doc/latest/](https://cassandra.apache.org/doc/latest/)
* DynamoDB Best Practices: [https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html)
## Example checklist before production rollout
* [ ] Implement idempotent writes (`entry_id` per follower).
* [ ] Add batching and worker sharding.
* [ ] Define TTL/retention policy and compaction strategy.
* [ ] Add monitoring for consumer lag, write failures, and hot keys.
* [ ] Design special handling for high-follower-count accounts.
* [ ] Test recovery and resume semantics for the consumer stream (offset management).
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.