Designing a Content Moderation Pipeline: Trust and Safety at Platform Scale

Content moderation is the system design question with asymmetric catastrophic errors in both directions: leave abusive content up and real people are harmed (plus regulators arrive with deadlines measured in hours); take legitimate content down and you've silenced someone wrongly at scale. Add a billion daily uploads, adversaries who actively probe your defenses, and human reviewers who are both the scarcest resource and the ones absorbing psychological harm — and you get a pipeline design where the routing of uncertainty is the whole game.

The tiered pipeline

No single mechanism survives the volume, so moderation is a funnel ordered by cost and confidence — the recommender funnel's dark twin:

upload -> HASH MATCHING (known-bad, ~exact)      -> block at ingest
       -> CLASSIFIERS (ML per harm type)          -> high conf: auto-action
                                                  -> uncertain: enqueue
       -> HUMAN REVIEW (queues, tiered expertise) -> decision + label
       -> APPEALS (second human, audit trail)     -> reversal signal

Tier 1 — hash matching catches known abusive material at ingest: cryptographic hashes for exact copies, perceptual hashes (PhotoDNA-class, resistant to resizing/re-encoding) for near-duplicates, shared industry hash sets for the worst categories (CSAM via NCMEC, terror content via GIFCT). This tier is cheap, precise, runs synchronously in the upload path, and is legally non-optional in several categories. Design notes: perceptual-hash matching is a nearest-neighbor problem in Hamming space (the SimHash machinery again), and the hash lists themselves are sensitive — distributed to edge matchers as encrypted blooms/indexes, never as raw material.

Tier 2 — classifiers, one per harm family (nudity, violence, hate, spam, self-harm), because harms have different signals, thresholds, and legal regimes. Each outputs a score feeding a three-way decision: auto-remove above a high-precision threshold, ignore below a low one, and — the design's center — route the uncertain middle to humans. Text models run inline; image/video models run near-line (seconds); expensive video understanding runs async with provisional availability. Virality changes the calculus: content accelerating toward 100K views earns re-scoring and priority review because its blast radius grew — the queue is risk-ordered, not FIFO.

Human review: the scarce, expensive, harmed resource

The review system is a work-distribution problem with constraints no other queue has. Prioritization by expected harm (severity × predicted reach × classifier uncertainty), not arrival time. Routing by specialization: language, market/legal context, harm type — a self-harm case needs a trained reviewer with wellness protocols, not a generalist. Quality control: golden tasks (known-answer items seeded into queues), double-review with adjudication for high-stakes categories, per-reviewer precision tracking. Reviewer welfare as an architectural requirement: exposure caps per shift for graphic categories, blurred/grayscale-first presentation with opt-in reveal, wellness rotations — these shape queue design as much as any throughput target, and mentioning them unprompted is the difference between having read about T&S and having thought about it.

Every human decision is a label: it feeds classifier retraining (the flywheel — human judgment converts to future machine coverage), threshold calibration, and — via appeals reversals — measurement of the false-positive rate that pure automation can't see about itself.

Appeals and the policy engine

Appeals are not a courtesy; they're the system's error-correction channel and increasingly a legal mandate (DSA-class regulation requires notice, reasons, and review). A second, independent reviewer sees the case with context (the policy clause cited, the classifier score, the first decision); reversals flow back as high-value training signals and as audit metrics per policy area. Everything is audit-logged immutably — who/what/why for every action — because regulators, courts, and transparency reports all consume it (and because "why was this removed" must be answerable years later).

The policy engine deserves explicit design: policies change weekly, so enforcement logic lives as versioned, declarative rules (policy X, clause Y, threshold Z per market) evaluated against classifier outputs and metadata — never hard-coded in the pipeline. Jurisdiction is a first-class dimension: the same content can be legal in one market and not another; actions are geo-scoped (blocked in market A, visible elsewhere), which means the enforcement state model is per-content × per-market, not a boolean.

Adversarial pressure closes the loop: evasion is constant (re-encoding to break hashes, leetspeak against text models, coordinated cross-account posting), so the system needs its own abuse telemetry — clusters of near-identical content from linked accounts (the fraud post's entity graph, repurposed), probing patterns against thresholds, and rapid-response hash listing when a novel harmful item starts spreading (crisis mode: minutes matter, virality curves don't wait for retraining).

Interview probe	Answer sketch
Why not block everything the classifier flags?	Precision at scale: 99% precision on a billion items = 10M wrongful actions/day — the uncertain band must route to humans
Live streams?	Hardest mode: sampled frames + audio classifiers + viewer reports feeding a priority interrupt queue; latency budget is seconds and mistakes are public
Reviewer disagreement?	Adjudication layer + policy clarification loop — disagreement clusters are policy bugs, routed to policy owners, not just re-rolled
Metrics?	Prevalence (what slipped through, measured by sampling), action rate, appeal reversal rate, time-to-action at percentiles — and never celebrate volume of removals alone

The distilled architecture: exact-match the known, classify the knowable, route uncertainty to prioritized humans, let appeals audit the machine, and version policy like code. The pipeline's real product isn't removals — it's defensible decisions at scale, each one reconstructable years later.