Designing a Feature Store and Model Serving Platform

Companies discover the need for a feature store the same way, every time: a model that crushed offline evaluation quietly underperforms in production, and three weeks of debugging reveals the training pipeline computed user_7day_purchases from the warehouse while the serving path computed it from a Redis counter with different semantics. Nothing was "broken." The two systems just disagreed about what a feature means. A feature store is the infrastructure that makes such disagreement impossible; model serving is its consumer. Together they're the ML-infra interview.

The core contract: define once, materialize twice

A feature is declared once — name, entity (user, item, pair), transformation, freshness SLO — and the platform materializes it into two stores with one definition:

feature: user_7day_purchase_count (entity: user_id)
  -> OFFLINE store: warehouse/lake tables, full history, timestamped
     (training: point-in-time joins over months of data)
  -> ONLINE store: KV (Redis/DynamoDB), latest value only
     (serving: p99 ~5ms lookups at request time)

Batch features compute in the warehouse and sync to the online store on schedule; streaming features compute once from the event stream and dual-materialize — the same aggregation writing both the online value and the offline history. Either way, the property being purchased is train/serve parity: one transformation definition, so the model meets the same number in production that it trained on.

Point-in-time correctness: the silent model-killer

Training data construction is where ML platforms quietly win or lose. A training row is "the features as they were at the moment of the label event" — joining today's feature values onto last month's impressions leaks the future into the past, and the model learns from information it won't have at serving time. Offline stores therefore keep timestamped feature history, and training-set generation is an as-of join (each label event picks the latest feature value before its timestamp). The belt-and-suspenders version, increasingly standard: log the exact feature vector used at serving time and train on those logs — parity by construction, at the cost of only learning about features you already serve (new features still need the as-of path to backfill history).

Model serving: an inference tier is a read path with a GPU bill

Serving infrastructure is a specialized stateless tier: models packaged as versioned immutable artifacts (the registry is "artifact store + metadata + lineage — which data, which code, which features"), loaded by serving processes that expose predict RPCs. The systems content:

• Latency mechanics: dynamic batching (coalesce concurrent requests into one forward pass — GPUs are throughput devices; batching trades ~1-5ms of queueing for 10× throughput), model warm-up before traffic, per-model resource isolation so one team's 2GB ranker doesn't evict another's fraud model.
• Rollouts: models are deploys — shadow mode first (new model scores live traffic, predictions logged, not acted on — the only honest offline-online bridge), then canary by traffic slice with the experimentation platform reading product metrics, not just model metrics; instant rollback = repoint to the previous artifact. Champion/challenger as a permanent posture.
• Degradation policy: the feature fetch precedes inference, so feature-store latency is inside the prediction SLO. Timeouts get feature defaults (score with population priors rather than fail the request) and the model must be trained with dropout on those features so degraded ≠ deranged. A checkout page never 500s because Redis hiccuped; it serves a slightly dumber fraud score and flags it.

Monitoring completes the loop, and it is distribution monitoring, not just uptime: input feature drift (today's traffic vs training distribution), prediction drift, and — lagging — label-based quality when ground truth arrives. Feature drift alarms are the smoke detector that fires before business metrics burn.

The org-shaped part

A feature store is also a catalog: discoverable, owned, documented features with lineage — because its second-order value is reuse (the fraud team's account_age_days shouldn't be reimplemented by ads with an off-by-timezone bug). Governance rides the same metadata: PII tagging, retention, "which models consume this feature" for impact analysis when a pipeline breaks upstream. In an interview, mentioning that the failure blast radius of one late feature pipeline is every model consuming it — and that the platform must answer that query — reads as having operated one.

Interview probe	Answer sketch
Why not compute features in the model service?	Duplicated logic per model, no history for training joins, no reuse — the store exists to own the definition
Online store goes down?	Feature defaults + degraded-mode scoring + circuit breaker; models trained to tolerate missing features — never inline-fail the product
Streaming feature, 5-second freshness, also needed in training?	Dual-materialize from one stream job; offline side appends timestamped values for as-of joins
GPU vs CPU serving?	Batch-friendly deep models → GPU with dynamic batching; small tree ensembles → CPU is cheaper and simpler — route by model, not by fashion

The distilled contract: one feature definition materialized to two stores, point-in-time joins (or served-feature logs) for training, versioned artifacts behind shadow→canary rollouts, and degradation that dumbs down instead of falling over. Everything else in MLOps is tooling around those four promises.