Designing a News Feed: Fan-Out on Write vs Fan-Out on Read

"Design the Twitter timeline" endures as an interview question because it hides a real tension: the data model that makes writing a post cheap makes reading a feed expensive, and vice versa. There is no clever schema that gets both — only a trade, and one famous edge case that forces you into a hybrid.

The setup: users follow users. A user posts; every follower should see it in their home feed, newest-ish first, within a few seconds. Reads dominate writes by a couple of orders of magnitude — a person refreshes their feed far more often than they post. Feeds are allowed to be slightly stale; nobody files a bug because a post took eight seconds to appear.

Fan-out on read: compute the feed when asked

The normalized approach. Posts live in a posts table; follows live in a follows table; a feed is a query:

SELECT p.* FROM posts p
JOIN follows f ON f.followee_id = p.author_id
WHERE f.follower_id = :me
ORDER BY p.created_at DESC
LIMIT 50;

Writes are perfect — one insert, visible instantly, delete is trivial. The read is the problem: for a user following 1,000 accounts, this is a 1,000-way merge executed on every refresh, and once posts are sharded by author, it becomes a scatter-gather across shards. The feed is the hottest read path in the product, and this design makes it the most expensive query in the system. It works fine at small scale and for follow graphs in the dozens; it collapses at consumer scale.

Fan-out on write: precompute every feed

Invert it. Materialize a feed per user — typically a Redis list or sorted set of post IDs, capped at the last ~800 entries. When someone posts, a fan-out worker looks up their followers and pushes the post ID into each follower's feed:

POST /tweet
  1. Insert post into storage (source of truth)
  2. Enqueue fanout job                      ← async, via queue
  3. Worker: for each of N followers:
       ZADD feed:{follower_id} {ts} {post_id}
       ZREMRANGEBYRANK feed:{follower_id} 0 -801   # cap length

Now the read path is beautiful: one Redis fetch of 50 IDs, hydrate the post bodies from a cache, render. Sub-10ms, embarrassingly scalable, and read cost no longer depends on how many people you follow.

You paid with three new problems. Write amplification: one post by a user with 5,000 followers is 5,000 feed writes. Deletes and edits: also amplified — you either chase the post ID through every feed or filter deleted IDs at read time (everyone does the latter). And storage: an 800-entry feed for hundreds of millions of users is real memory, which is why feeds are capped and only allocated for recently active users.

The celebrity problem

Then someone with 100 million followers posts. Fan-out on write now means 100 million Redis writes per tweet — minutes of queue backlog, during which followers see the post at wildly different times, and one hyperactive celebrity can saturate your fan-out workers for everyone. This is the hot key problem wearing sunglasses: any design that does per-follower work on write is quadratic in exactly the accounts that generate the most engagement.

So nobody runs pure push. The production answer is the hybrid:

if author.followers < THRESHOLD:      # ~10k-ish
    fan out on write                   # cheap, posts are rare, followers few
else:                                  # celebrities
    write once to author's own list   # no fan-out at all

At read time, merge the two sources: the user's precomputed feed (push) plus fresh posts from the handful of celebrities they follow (pull). A typical user follows single-digit celebrity accounts, so the pull half is a merge of a few short, heavily cached lists — nothing like the 1,000-way merge of pure pull. Instagram and Twitter have both described variants of exactly this split.

The same lens generalizes: push when the audience is small and reads are frequent; pull when the audience is enormous or the reader is inactive (fan-out to a user who has not opened the app in a year is wasted work — many systems skip fan-out for dormant users and rebuild their feed on return).

Ranking, pagination, and the details that bite

Ranking. Modern feeds are not chronological; a scoring model reorders candidates. Architecturally this slots in cleanly: the fan-out machinery becomes candidate generation, and at read time a ranking service scores ~500 candidates and returns the top 50. The push/pull decision is unchanged — you still need candidates cheaply.

Pagination. Offset pagination breaks immediately: new posts arrive between page 1 and page 2 and everything shifts. Feeds use cursors — "give me 50 items older than post ID X" — which is also why time-sortable post IDs matter: the ID doubles as the cursor.

Consistency. A feed is allowed to be seconds stale, but a user must always see their own post immediately — read-your-writes for the author, eventual for everyone else. Cheap trick: inject the author's own recent posts at read time regardless of fan-out progress.

Failure handling. Fan-out must be at-least-once with idempotent writes — a sorted-set add of the same post ID twice is naturally idempotent, which is one quiet reason ZSETs beat lists here.

Takeaways

Feed design is one decision wearing many costumes: do work on write (push) or on read (pull)? Push wins when reads dominate and audiences are modest; pull wins for celebrities and dormant users; every system at scale runs the hybrid with a follower-count threshold. Cap materialized feeds, use ID-based cursors, filter deletes at read time, and give authors read-your-writes. Say "fan-out" three times in an interview and the interviewer relaxes — demonstrate you know why the threshold exists and they start taking notes.