Designing Recommendation Infrastructure: Candidates, Ranking, and the 100ms Funnel

Every recommendation question — TikTok's feed, Netflix's rows, Amazon's "you might like" — reduces to the same impossible arithmetic: a billion candidate items, a user waiting ~200ms, and a ranking model that costs ~1ms per item scored. You are seven orders of magnitude short. The entire architecture of recommender infrastructure is one idea applied repeatedly: spend cheap compute to shrink the candidate set before spending expensive compute to order it. The ML gets the papers; the funnel is the system.

The funnel

~10^9 items -> RETRIEVAL (cheap, parallel)  -> ~10^4 candidates
           -> RANKING   (real model)        -> ~10^2 scored
           -> RE-RANKING (business logic)   -> ~10 shown

Retrieval runs several candidate generators in parallel, each embodying one "reason to recommend": collaborative signals (users like you watched X), content similarity, social graph, freshness pools, subscribed/followed sources. Union their outputs. The dominant retrieval machinery today is the two-tower model: a user tower and an item tower trained so relevant pairs land near each other in embedding space — item embeddings precomputed offline into an ANN index (HNSW graphs or quantized IVF — the same vector-search machinery as RAG, at recommender scale), user embedding computed per request, nearest-neighbor lookup returns thousands of candidates in single-digit milliseconds. The two-tower split is a systems decision disguised as modeling: no user-item interaction terms in retrieval means item vectors are cacheable and the per-request cost is one tower plus one ANN query.

Ranking applies the heavy model (hundreds of features, cross-features, sequence models) to the ~10K survivors — often split again into a light ranker (10K → 500) and heavy ranker (500 → 100), same funnel logic fractally. Re-ranking is where product policy lives: diversity injection (don't show 10 videos from one creator), freshness boosts, deduplication against recent impressions, business rules and integrity filters. Keeping policy out of the model and in this explicit layer is what keeps launches debuggable and experiments interpretable.

The feature platform: where correctness lives

Ranking models consume features from three temperatures — static item metadata, batch-computed aggregates (30-day CTR per item), and real-time signals (clicks in the last five minutes, current session trajectory) — served from a low-latency feature store keyed by user/item. Two systemic failure modes dominate this layer, and naming them is senior signal:

• Training-serving skew: the model trained on features computed one way (batch SQL) and serves on features computed another (online service); silent distribution drift, mysterious quality loss. Cure: one feature definition, two materializations — the feature-store pattern — plus logging the features actually used at serving time and training on those logs, never on reconstructions.
• Point-in-time leakage: training data must join features as they were at impression time, not as they are now. Post-hoc joins leak the future into training, models look great offline, then flop online — the classic "offline AUC up, online metrics flat" mystery.

Real-time features ride the standard stream stack (events → Kafka → aggregation → feature store) with staleness SLOs per feature: session features tolerate seconds; a "user just blocked this creator" signal must apply now — the integrity-flavored features get the freshest path.

Closing the loop, and the loop's disease

Impressions and engagements flow back as training data (the ad-click pipeline again — dedupe, exactly-once-ish, into the lake), models retrain on cadences from weekly (heavy rankers) to near-continuous (lightweight online updates). The loop has a structural disease worth volunteering: feedback bias. The model only learns from what it chose to show — items never surfaced never earn signals, positions bias clicks, and the system converges on self-fulfilling popularity. Countermeasures are architectural, not incidental: reserved exploration traffic (a small % of slots serve stochastic/fresh candidates), position-debiasing in training, and the experimentation platform's holdouts measuring whether the loop is optimizing the metric or just optimizing itself. Cold-start gets the same treatment — content-based retrieval and explore slots are the on-ramp for new items and users.

Serving-side operations mirror any high-QPS read system: candidate generators fan out with hedged requests and per-stage timeouts (a slow generator gets dropped, not waited for — graceful degradation is "fewer candidate sources," invisible to the user), embedding/ANN indexes rebuild on schedules with atomic swap (the immutable-index-ship pattern from search/autocomplete), and the whole funnel is instrumented per-stage because "recommendations got worse" must decompose into which stage changed.

Interview probe	Answer sketch
Why not one end-to-end model?	The funnel is the latency budget; every stage exists because the next is 100× more expensive per item
Feature store consistency?	Same definition batch+stream, log-and-train-on-served-features, point-in-time joins — skew is the #1 silent killer
Real-time personalization ("responds to my last click")?	Session features via stream into the store + lightweight re-ranking online; retrain ≠ respond
How do you know retrieval is the weak stage?	Stage-wise recall metrics: did the item the user eventually engaged exist in the candidate set? — instrument the funnel, not just the end

The reusable shape: a compute funnel ordered by cost-per-item, a feature platform whose whole job is train/serve parity, and a feedback loop with deliberate exploration so the system measures the world instead of its own reflection.