Designing a Durable Workflow Engine: How Temporal Makes Code Survive Crashes

Every backend eventually accumulates the same swamp: multi-step business processes — charge the card, reserve inventory, email the customer, wait three days, maybe refund — implemented as queues feeding state machines feeding cron jobs feeding retry tables. Each step can crash, each retry can double-fire, and the actual business logic is smeared across five services and a status column. Temporal-style durable execution makes a radical promise: write it as one ordinary function, and the platform guarantees it runs to completion — across crashes, deploys, and weeks of wall-clock time. Designing that engine is a beautiful interview because the API sounds like magic and the implementation is one honest idea.

The one idea: event sourcing + deterministic replay

The engine never persists a process's memory. It persists the workflow's event history — every input and every result that crossed the workflow's boundary:

WorkflowStarted{order}
ActivityScheduled{charge_card} -> ActivityCompleted{txn_id}
TimerStarted{3 days}           -> TimerFired
ActivityScheduled{send_email}  -> ActivityFailed{timeout} -> ActivityCompleted{retry#2}

When a worker crashes mid-workflow, another worker rebuilds the exact state by re-executing the workflow function from the top, feeding it recorded results instead of performing effects again: the code calls charge_card(), the replayer sees ActivityCompleted{txn_id} in history and returns it instantly; execution races forward to the first command not in history, and only that executes for real. The function's local variables, loops, and if-branches are the state — reconstructed by determinism rather than serialized.

The price is the determinism contract: workflow code must make identical decisions on replay. No wall-clock reads, no random, no direct I/O, no iteration over unordered maps, no threads — all such effects go through the SDK (workflow.now(), workflow.random(), activities), which records them into history so replay can feed them back. This is the same determinism discipline as the matching engine and every replicated state machine — durable workflows are event sourcing applied to program counters. Code upgrades get the same treatment: histories are replayed by the code version that wrote them (version markers/patching), because replaying old history through new logic is divergence by another name.

Activities: where effects live

Everything non-deterministic — HTTP calls, DB writes, sending email — is an activity: executed outside replay, at-least-once, with results recorded into history. The engine owns retries (policy in code: backoff, max attempts), timeouts (schedule-to-start, start-to-close, heartbeat — long activities heartbeat so a dead worker is detected mid-task, not at timeout), and the corollary every candidate must say unprompted: activities must be idempotent, because at-least-once is the delivery guarantee and the idempotency key (workflow ID + activity ID) is handed to you. Exactly-once workflow decisions, at-least-once effects, idempotency bridging the two — the same theorem as everywhere else, given ergonomics.

Timers are just events — sleep(30 days) writes TimerStarted and the worker releases the thread; the engine's durable timer service (sharded timer wheels over the same persistence) fires it a month later into the workflow's task queue. A million workflows mid-sleep cost storage rows, not threads. Signals (external events into a running workflow) and queries (read its state) ride the same history mechanism.

The architecture underneath

history service:  owns event histories + workflow mutable state
                  (sharded by workflow ID, each shard a consistent log)
matching service: task queues — "this workflow has a decision to make",
                  "this activity needs running" -> long-polled by workers
workers (yours):  run workflow code (replay) + activities; stateless, scale at will

Two design points carry the throughput story. Shard by workflow ID: each workflow is a serial stream of decisions (single-writer per workflow — no intra-workflow races by construction), while millions of workflows spread across shards; per-workflow throughput is deliberately bounded (~tens of events/sec), and workloads needing more must split workflows (the documented "one workflow per entity" pattern). Sticky execution: replaying a 10K-event history per decision would be brutal, so the matching layer routes a workflow's next decision to the worker that already has it cached in memory; replay-from-scratch is the recovery path, not the common path — plus continue-as-new to truncate unbounded histories (the log-compaction move).

Failure semantics worth stating precisely: worker dies → task times out → redelivered elsewhere → replay reconstructs → continue. History service node dies → shard fails over (the history log is replicated — Raft or the datastore's own replication). The user's code sees none of it; "durable execution" means the program is the replicated state machine.

Interview probe	Answer sketch
vs a saga orchestrator / step functions DSL?	Same pattern (orchestration + compensation); durable execution's bet is expressing it in a real language — loops, try/catch as compensation, local variables as state — instead of JSON state machines
Workflow does if now().hour < 12?	Determinism violation — replay at 3pm diverges; SDK's now() records time into history so replay agrees
Poison workflow (bug makes it crash every replay)?	Deploys don't help old histories — versioning/patching APIs, or terminate-and-migrate; this is the operational sharp edge
Why long-poll task queues, not push?	Workers are behind NAT/firewalls, scale elastically, and pull-based matching gives natural backpressure + sticky routing

The distilled shape: record everything that crosses the boundary, demand determinism inside it, replay to recover — and then spend the entire rest of the system (sharded history, timers, sticky caches, versioning) making that one idea cheap enough to be an everyday tool.