Designing an Online Judge: Sandboxed Code Execution at LeetCode Scale

Every system design interview worries about malicious input. An online judge is the only mainstream system whose core feature is executing malicious code — arbitrary programs from anonymous strangers, thousands per minute, on your hardware. while(true) fork(), 8GB allocations, attempts to read /etc/passwd, sockets back to a C2 server: all of it arrives labeled "solution.py," and your job is to run it, meter it, and grade it. Security people spend careers preventing remote code execution; this product is remote code execution.

Requirements

Submit code in ~20 languages; run against hidden test cases with time/memory limits; verdict (AC/WA/TLE/MLE/RE) in a few seconds; absolute isolation between submissions and from the host; contest mode — 100x spike in minute one, fairness and consistent timing throughout.

Architecture: queue between trust boundaries

The shape is asynchronous by necessity: execution takes seconds and can be adversarially slow, so nothing user-facing waits synchronously on it.

web/API -> submissions DB -> queue (per-language or weighted)
        -> judge workers (isolated fleet) -> verdict events -> DB/WebSocket

The queue is also the security seam: the web tier never executes anything; workers never face the internet. The worker fleet runs in its own network segment with no egress and no credentials worth stealing — design for the sandbox to fail someday, and make the blast radius a machine you can reimage without caring.

The sandbox: layers, because one will fail

The interview's center of gravity. A single mechanism is a single CVE away from compromise; production judges stack:

1. Resource limits — cgroups for CPU time, memory, PIDs (fork bombs die at pids.max=64), plus wall-clock watchdog (a sleeping process burns no CPU but must still TLE).
2. Namespaces — mount (private minimal filesystem, read-only toolchain), network (none), PID (can't see or signal other processes), user (root inside = nobody outside).
3. Syscall filtering — seccomp-bpf allowlist: read/write/mmap/exit and little else; no socket, no ptrace, no mount. The allowlist is per-language runtime (JVMs legitimately need more than a static C binary — this is where judge maintainers earn their pay).
4. MicroVM or gVisor for the paranoid tier — a stripped kernel-boundary (Firecracker boots in ~125ms) so a container-escape zero-day still lands inside a disposable VM.

Layers 1-3 are what open-source judges (isolate, the IOI sandbox, judge0) implement; layer 4 is what you add when the workload is fully anonymous. And regardless of layers: workers are cattle — recycled after N submissions or any anomaly, rebuilt from image. Assume compromise; make it worthless and brief.

Fair timing: the subtle correctness problem

A verdict of 1.98s vs 2.01s changes AC to TLE, so measurement must be reproducible: identical hardware class per queue (or per-language time multipliers calibrated per machine type), one submission per core with pinning (no noisy neighbors on the judging core), CPU-time as the primary limit with wall-time as backstop, and compile time metered separately from run time (compilation is also sandboxed — #include bombs and template-metaprogramming explosions are attacks on the compiler). Contest platforms rerun near-limit submissions to damp scheduling jitter; determinism is a fairness feature you architect for, not a benchmarking nicety.

Grading without leaking secrets

Test cases are IP — leaking them via side channels is a classic attack (print the input, time your own failures, exfiltrate through the verdict bit). Mitigations: the submitted process reads stdin and writes stdout only, with the runner (outside the sandbox) streaming test data in and comparing out; per-test verdicts truncated on hidden tests; output size caps (a 10GB stdout is an attack on the judge). Comparison itself ranges from byte-diff to tolerance-aware numeric compare to a special judge — a problem-specific checker binary for multiple-valid-answer problems, which is itself semi-trusted code and gets a sandbox too. Everything downstream of "run" is idempotent: a worker dying mid-judgment just means the submission re-queues and re-runs — verdicts are deterministic, so at-least-once execution is safe by construction.

Contest mode: the load story

Weekly contest, minute zero: tens of thousands of submissions for the same problem. This is a shaped spike — you know when it starts — so: pre-scale the worker fleet on schedule, per-user submission rate limits (also anti-brute-force), queue with per-user fairness (one user's 50 resubmits don't starve others), and verdicts pushed over WebSocket so nobody polls. Priorities matter: contest queue > practice queue; within contest, FIFO for fairness — and publish that policy, because in competitive programming the scheduler is part of the rules.

Interview probe	Answer sketch
Docker enough?	Containers share the host kernel — one kernel bug from escape; add seccomp + user-ns, or microVMs for anonymous code
Infinite loop with no CPU (sleep)?	Wall-clock watchdog kills it; CPU-time alone is insufficient
Judge a submission twice = two verdicts?	Never — deterministic env + fixed seeds; nondeterminism is a judge bug, and reruns exploit it
Language versions drift?	Toolchains as versioned immutable images; a problem's limits are calibrated against a pinned image set

The transferable frame: this is defense-in-depth as a product architecture — trust boundaries drawn so that every component assumes its neighbor is compromised, resources metered as first-class semantics rather than ops hygiene, and determinism treated as a fairness contract. Most systems hope they never run attacker code; this one schedules it.