Two-Phase Commit: Why Distributed Transactions Stall

You have two writes that must both happen or neither happen: an order row and a payment capture, a ledger line and a queue message, a row on shard A and a row on shard B. A local BEGIN/COMMIT cannot wrap them. The textbook protocol for this is two-phase commit (2PC). It does give you atomicity across resources. The price is that a crash at the wrong moment leaves every participant holding locks, waiting for a coordinator that is not there.

This is the protocol the outbox, saga, Percolator, and Spanner posts all assume you can picture. It is worth its own post, because the failure mode is not "it is slow." The failure mode is stall: the system is not wrong, it is stuck.

The two phases

A coordinator drives the transaction. The participants are the resources that must agree. The protocol is a vote, then a decision.

Phase 1 — prepare
  coordinator → all participants: PREPARE
  each participant:
    flush its work to stable storage  (a WAL write)
    if it can commit: vote YES, hold locks, wait
    if it cannot:     vote NO

Phase 2 — decide
  if every vote is YES:
    coordinator writes COMMIT to its own log     ← the decision
    coordinator → all participants: COMMIT
  else:
    coordinator writes ABORT to its own log
    coordinator → all participants: ABORT
  participants apply, release locks, ACK

Until a participant votes YES, it can still abort on its own. After a YES, it has promised: it will not unilaterally commit, and it will not unilaterally abort. It holds the locks that protect that promise until the coordinator names the outcome. That is the whole protocol. Atomicity comes from the promise plus the coordinator's single logged decision.

The coordinator's log record is the commit point. If the coordinator crashes before that write, recovery treats the transaction as aborted — no participant was told to commit, and any YES-voter is told ABORT when the coordinator returns. If it crashes after that write, recovery re-sends COMMIT. Participants must be able to receive the same decision twice; the second COMMIT is a no-op. That is why participant prepare has to be a durable WAL write, not a memory flag: a restarted participant has to remember it voted YES.

XA, the interface most databases expose for this, is the same diagram with different nouns: xa_prepare, xa_commit, xa_rollback. The application (or a transaction manager) is the coordinator. The databases are participants.

Where it stalls

The dangerous window is after a participant has voted YES and before it has heard the decision.

P1: YES, locks held
P2: YES, locks held
coordinator: crashed, decision not yet written — or written, not yet sent

P1 and P2 cannot finish. They promised. Asking them to time out and guess is how you get a split brain: one participant commits, the other aborts, and the "atomic" transaction is two different outcomes. So they wait. Every other transaction that needs those rows waits with them. The coordinator is the only process allowed to break the tie, and it is gone.

That is the blocking property. 2PC is live only while the coordinator is live during that window. A participant that voted YES and then died is the same shape from the other side: the coordinator cannot finish phase 2 until that participant returns, because COMMIT has to reach every YES-voter. Availability of the transaction is the availability of every participant plus the coordinator, for the whole of phase 2.

Heuristic decisions — "if the coordinator is silent for 30 seconds, abort" — are how operators get unstuck and how they get inconsistent. They are a manual override, not a protocol. Once a participant has voted YES, a local timeout is a guess about a decision that may already be COMMIT on the coordinator's disk.

Three phases does not save you

Three-phase commit (3PC) inserts a pre-commit round between prepare and commit. The extra round is there so that, after the coordinator has told everyone "we are going to commit," a surviving participant can be sure that some other participant already knows the decision, and a new coordinator can be elected without blocking.

It removes the "coordinator vanished, everyone holds locks forever" case under a fail-stop model where messages are reliable. It does not survive a network partition. One side can elect a coordinator, see a pre-commit, and commit. The other side never got pre-commit, elects its own coordinator, and aborts. You traded a stall for a split. Partitions are the failure mode production systems actually have, so 3PC stayed a textbook curiosity. Almost nobody runs it.

The lesson is not "add a round." The lesson is that non-blocking atomic commit in an asynchronous network is not something you get by decorating 2PC. You change the assumption: replicate the coordinator so it cannot disappear, or stop asking two independent systems to commit as one.

What you run instead

Most application-level "distributed transactions" should not be 2PC.

Keep the transaction inside one database. If the two writes can live in one engine, they should. Cross-resource atomicity is a last resort, not a default. Shard keys exist in part so that a business transaction stays on one shard; the sharding post's rule that transactional units travel together is this paragraph in operational form.

Database plus message: use an outbox. Write the business row and an outbox row in the same local transaction. A relay publishes from the outbox. You get "the event exists if and only if the row exists" without asking the broker to speak XA. Brokers mostly do not, and the ones that do couple your commit path to the broker's availability — the same blocking shape, with worse operational answers.

Work that already crossed service boundaries: use a saga. Each service commits locally. A later step compensates if a later step fails. You have given up atomic isolation across the whole business process; you have kept each service independently available. That is the right trade for order/pay/ship. It is the wrong trade for "these two rows in one ledger must never diverge," which is why ledgers stay in one database.

When you truly need distributed ACID, replicate the coordinator. That is the Spanner move: every 2PC participant is a Paxos (or Raft) group, and the coordinator is a group too. Classic 2PC's fatal case — "the coordinator process died" — no longer exists, because a process is not the coordinator; a quorum is. Percolator makes the same move without a dedicated transaction service: the commit flag is a cell in the data store, and any client can finish a crashed transaction by reading it. Both designs keep 2PC's vote then decide shape. They delete the single process that made the shape stall.

When 2PC is still the right tool

Inside one distributed database, with a replicated coordinator, 2PC is how cross-shard writes become a single commit. CockroachDB, Spanner, and TiDB all look like that. You are not coordinating a payment processor and a Kafka cluster. You are coordinating two ranges that already share a log format, a clock story, and an operational team.

Two resources in the same process that both speak XA — a rare application server talking to two databases it owns — can still use a transaction manager. Measure the window you are willing to be blocked, and have a runbook for heuristic decisions that does not pretend they are consistent. If you cannot name that runbook, you do not want 2PC there.

What you should not do is reach for 2PC because two microservices both need to "be in the same transaction." They are not. They have two failure domains. 2PC will join those domains: an outage in either, or in the coordinator, becomes an outage in both. That is the opposite of why you split the services.

The takeaway

Two-phase commit is a vote, a durable decision, and a promise that YES-voters will wait. Atomicity is real. Liveness is borrowed from the coordinator. A crash in the window after prepare turns that borrow into a stall, and a timeout in that window turns it into a split. Three-phase commit spends another round and still splits under a partition. The designs that actually shipped — outbox, saga, Percolator, Spanner — either refused to span two systems, or they kept 2PC and made the coordinator a replicated fact instead of a process. Use 2PC where the coordinator cannot disappear. Everywhere else, stop asking two resources to share a commit.