Designing Real-Time Fraud Detection: Scoring Inside the Payment Path

Fraud detection is the rare ML system that sits inline — inside the payment authorization path, where its latency budget is measured against a card network timeout and its false positives are declined groceries. Block too little and you eat chargebacks; block too much and you lose customers cheaper than the fraud was. The design problem is threefold: compute rich features in milliseconds, combine machine judgment with human-written rules, and learn from labels that arrive weeks late. It's the payments interview and the ML-infra interview shaken together, which is why banks and marketplaces love asking it.

The inline constraint shapes everything

authorize(txn) -> [fraud scoring: ~50-80ms budget] -> approve / decline / challenge

The decision is synchronous — money is about to move — so the entire apparatus must fit a two-digit-millisecond budget with a fallback for its own failure. That single constraint dictates the architecture: features must be precomputed or O(1)-computable, models must be served hot, and the system must degrade to something safe (conservative rules-only scoring) rather than time out, because "fraud system down" cannot mean either "approve everything" or "decline everything."

Features: three temperatures, one store

The signal in fraud is overwhelmingly behavioral aggregates — is this transaction weird for this card, this merchant, this device?

• Batch (nightly): account age, historical spend percentiles, merchant category profiles.
• Streaming (seconds): transaction count/sum per card in last 5min/1h/24h, distinct merchants per hour, distance from previous transaction ("card in two countries in one hour" is a velocity feature, not a rule). Maintained by a stream job as sliding-window aggregates in the online feature store — count-min sketches and windowed counters doing their quiet work at scale.
• Request-time: device fingerprint match, IP/geo vs billing, session signals — computed at the edge, joined in.

The velocity features are the crown jewels, and their freshness is a correctness property: a fraudster's tenth transaction this minute must see features reflecting the first nine — which means the feature update path (auth event → stream job → online store) has an SLO of ~1-2 seconds, and the design conversation about "what if the stream lags" (score with stale velocity + widen the challenge band) is exactly the kind of failure-mode reasoning the interviewer wants.

Rules AND models: the two-lane decision engine

Every mature fraud stack runs both, because they fail differently. Models (gradient-boosted trees historically, deep/sequence models now) capture subtle interactions but retrain on a cadence — they're weeks behind a novel attack pattern. Rules are crude but deploy in minutes: when a fraud ring starts hammering gift cards from one BIN range at 3am, the on-call analyst ships IF mcc=gift_card AND bin IN (...) AND amount > X THEN challenge while the model is still blind to it. Rules are the fraud team's incident response; the model is their steady state. The engine evaluates both and combines: rules can hard-override (sanctioned country → decline, period), otherwise the model score feeds a decision ladder:

score < t1: approve   |  t1..t2: approve + monitor  |  t2..t3: CHALLENGE (3DS/OTP)
score > t3: decline   |  thresholds per segment, tuned to $-cost curves

The challenge band is the design's pressure valve — step-up authentication converts a coin-flip decision into evidence (a real user passes the OTP; a bot doesn't), buying precision exactly where the model is least sure. Thresholds are business dials set on explicit cost curves (chargeback cost vs decline cost vs friction cost per segment), reviewed as the fraud mix shifts — and every score + feature vector is logged for the case-review tooling where human investigators generate tomorrow's labels.

Learning with late labels

The ugly truth of fraud ML: ground truth arrives as chargebacks, 2-8 weeks later, plus sparse early signals (customer reports, manual review verdicts, issuer alerts). The training pipeline therefore has label-maturity discipline — train on transactions old enough for labels to have settled, weight early-signal labels differently — and the serving system leans on the delayed loop's antidotes: unsupervised/graph signals for novelty (this device links to five previously-confirmed-fraud accounts — entity-graph features are among the strongest and demand their own linkage store), analyst rules as the fast loop, and champion/challenger models in shadow to validate on live traffic before promotion. Adversaries adapt to whatever you deploy, so drift monitoring here isn't hygiene — attack traffic actively probes thresholds (card testing: $1 auths in bursts to find the line), and the system should detect probing as a pattern itself.

Interview probe	Answer sketch
Why not async scoring after approval?	Money's gone; async works only for slower rails (ACH holds) — card auth is the synchronous case by definition
Feature store misses SLA mid-decision?	Default features + rules-only floor + widened challenge band — degrade toward friction, not toward open gates
Precision/recall target?	Wrong framing — it's a cost function in dollars per segment; the ladder's thresholds are where finance and ML meet
Fraud ring with fresh cards, fresh devices?	Entity graph (shared IPs, addresses, device farms) + velocity at merchant and BIN grain — individual novelty, collective pattern

The transferable shape: an inline scorer with a strict budget and a safe floor, features at three freshness tiers, a rules lane for human-speed response beside a model lane for scale, a challenge band to buy evidence under uncertainty, and a training loop honest about label lag. That pattern — not the model — is what stops the bleeding at 3am.