Designing an Experimentation Platform: Assignment, Exposure, and the Stats Engine

Every serious product company converges on the same discovery: the hard part of A/B testing isn't statistics — it's plumbing. Assignment must be deterministic across a fleet of stateless services, exposure must be logged exactly where the experience diverged, hundreds of experiments must coexist without trampling each other, and the metrics pipeline must be trusted enough that a red number kills a launch nobody wants to kill. That's a distributed systems problem with a t-test on top, which is why "design an experimentation platform" has become a staff-level favorite.

Assignment: a hash, not a database

The rookie design stores assignments in a table — user 42 → variant B — and pays a lookup on every request, plus a consistency problem across services. The production design is stateless deterministic bucketing:

bucket = hash(experiment_salt + unit_id) mod 10000
variant = range lookup (0-4999: control, 5000-9999: treatment)

Any service, any time, computes the same answer from config alone — no assignment store, no lookup latency, consistent across web/mobile/backend by construction. The unit is a design decision with teeth: user ID for logged-in surfaces; device/cookie ID pre-login (with the identity-merge problem at login — pick a policy, it's product-visible); and session or request units only for stateless changes where a user flip-flopping between variants doesn't poison the experience. The salt per experiment decorrelates bucketings — without it, user 42 lands in treatment for every experiment and your "random" groups share a population.

Config (experiment definitions, allocations, targeting rules) distributes like any hot config: versioned snapshots pushed/pulled to every edge with second-scale propagation, evaluated locally in an SDK. Ramping (1% → 5% → 50%) grows the allocated bucket ranges monotonically so nobody assigned to treatment ever silently reverts.

Exposure: the log that is the experiment

The subtle correctness core: analysis compares exposed populations, so log the exposure event at the moment the code path actually diverged — not at assignment, not at page load. Log it too early (user assigned but never reached the feature) and you dilute the effect toward zero; log it conditionally in ways that correlate with the treatment (only treatment users hit the new code path that logs) and you've biased the sample — the dreaded sample ratio mismatch (SRM). A 50/50 experiment showing 50.4/49.6 with millions of users is not noise; it's a bug in exposure or assignment, and mature platforms run automated SRM checks that quarantine experiments before anyone reads their results. Exposure events flow through the standard analytics pipeline (dedup on (experiment, unit, first-exposure), at-least-once + idempotent merge — the ad-click machinery reused verbatim).

Overlap: layers, not locks

Hundreds of concurrent experiments can't each claim disjoint users — you'd cap at a handful. The standard answer is layered/orthogonal namespaces (Google's Overlapping Experiments paper): each layer owns one product surface (ranking, UI chrome, pricing); experiments within a layer split traffic exclusively; across layers, independent salts make assignments statistically orthogonal, so every user is simultaneously in one experiment per layer and interactions average out. Truly conflicting features (two experiments touching the same button) go in the same layer, or into an explicit mutual-exclusion group. Holdouts get a reserved slice — a percent of users exempted from everything for a quarter — measuring cumulative impact and catching the "wins that don't add up" problem.

The stats engine and guardrails

Metrics compute from exposure ⨝ events — typically overnight batch plus a near-real-time path for early warning. The platform's job is to make the defaults statistically safe: variance-reduction (CUPED — regress against pre-experiment behavior — routinely cuts required sample sizes 30-50%), sequential testing or fixed-horizon discipline (continuous peeking at naive p-values manufactures false winners — either always-valid p-values or locked readout dates), and multiple-comparison honesty when twenty metrics get eyeballed. Guardrail metrics — latency, crash rate, revenue floor — are monitored continuously with automatic shutoff wired to the config system: a treatment tanking p99 gets ramped to zero by a machine, minutes after the regression, before a human reads a dashboard. That kill switch is the platform's single most valuable feature and the reason it must own config distribution end to end.

Interview probe	Answer sketch
Why not assign at the load balancer?	Assignment must be unit-consistent across surfaces and services, not per-request at one hop — hash-on-unit in SDKs everywhere
User in treatment says feature broke — reproduce?	Deterministic bucketing means support can recompute their variant from config history; config snapshots are versioned and immutable per timestamp
Experiment on a two-sided marketplace?	Unit contamination (treated sellers affect control buyers) — cluster/geo randomization or switchback designs; saying "interference violates SUTVA" is the senior unlock
p-value dashboard updating live?	Sequential-testing corrections or it's a false-positive machine — the platform enforces the statistics, not the analyst

The shape to remember: deterministic assignment from distributed config, exposure logging at the divergence point, orthogonal layers for scale, and a stats engine with automated guardrails. The statistics are table stakes; the platform is what makes ten thousand people run them without a statistician in the room.