Database Replication: Leader-Follower, Quorums, and the Lag Nobody Budgets For

Every production database you have ever used is replicated, and almost every painful incident I have seen around databases traces back to a replication decision someone made without realizing it was a decision. Async replication that silently lost the last two seconds of writes during failover. A read replica that served ten-minute-old data to a payments dashboard. A "highly available" cluster that went read-only because one node in three died.

Replication answers three questions at once — how do we survive a node dying, how do we scale reads, and how much data are we willing to lose — and the mechanisms that answer them pull against each other. Here is the map.

Leader-follower: the default everyone starts with

One node (the leader) accepts all writes. It appends each write to its replication log — the write-ahead log in Postgres, the binlog in MySQL — and streams that log to followers, which replay it in order. Reads can go to any node; writes go to exactly one.

The single decision that defines this setup is when the leader acknowledges the client:

Mode	Leader confirms after	Write latency	Data loss on leader crash
Synchronous	Follower confirms it has the write	Slowest, coupled to follower health	None
Asynchronous	Writing locally only	Fastest	Everything not yet replicated
Semi-synchronous	At least one of N followers confirms	Middle ground	None, if the confirming follower survives

Fully synchronous replication to every follower sounds safe until one follower has a bad disk and every write in your system now waits on it. That is why nobody runs it. Fully async is what most MySQL and Postgres deployments run by default, and it means a leader crash loses whatever the followers had not received yet — usually milliseconds of writes, sometimes seconds under load, which is exactly when crashes happen.

Semi-sync is the pragmatic middle: one acknowledged copy elsewhere before you tell the client "committed." If your data matters, this is the floor.

Replication lag and the reads that lie

Followers replay the log with some delay — the replication lag. Normally tens of milliseconds; under a bulk import, a schema migration, or a follower doing a long vacuum, it can grow to minutes. Three user-visible anomalies fall out of this:

Read-your-writes violations. A user updates their profile, the write goes to the leader, the next page load reads a lagging follower, and their change is gone. Standard fixes: route a user's reads to the leader for a short window after they write, or pass a session token carrying the last-seen log position and only serve from replicas that have caught up to it.

Monotonic read violations. Two consecutive reads hit two different replicas with different lag, and time appears to move backwards — a comment exists, then doesn't. Pinning a session to one replica fixes it.

Cross-entity weirdness. An answer replicates before the question it responds to. This needs consistent-prefix guarantees, which per-partition ordering usually provides.

The engineering takeaway: replication lag is not an edge case, it is a property of your system that product behavior depends on. Measure it, alert on it, and decide explicitly which reads are allowed to be stale.

Failover: where the real danger lives

When the leader dies, some process must notice (timeouts), pick the most up-to-date follower, promote it, and repoint clients. Every step hides a trap:

• Lost writes. With async replication, the promoted follower is missing the old leader's final writes. If the old leader comes back, its extra writes conflict with the new timeline and are usually discarded. GitHub's well-known 2012 incident involved exactly this class of problem.
• Split brain. The old leader was not dead, just partitioned. Now two nodes accept writes. Without fencing — forcibly ensuring the old leader cannot commit — you get divergent data that no tool can automatically merge.
• Trigger-happy failover. If your detection timeout is too aggressive, a load spike looks like a death, and the failover itself causes the outage.

This is why "we have automatic failover" should prompt the follow-up question: what does it do with unreplicated writes, and how does it fence the old leader? If nobody knows, the answer is "loses them" and "it doesn't."

Multi-leader and the conflict tax

Multi-leader replication — writes accepted in multiple regions, replicated asynchronously to each other — buys you local write latency and surviving a whole-region outage. The price is that two leaders can accept conflicting writes to the same row, and someone must resolve that.

Last-write-wins resolution silently discards one of the writes, with "last" decided by clocks that disagree across machines. CRDTs and application-level merge logic are principled but expensive to build. My honest advice: avoid multi-leader unless geography forces it, and if it does, partition your data so each record has a single home region and cross-region conflicts become structurally impossible.

Leaderless: quorums instead of a boss

Dynamo-style systems (Cassandra, Riak, DynamoDB's internals) drop the leader entirely. Every write goes to N replicas; the client waits for W acknowledgements. Every read queries N and waits for R responses, taking the newest version. The invariant that makes reads see the latest write:

R + W > N        # read and write sets must overlap

N=3, W=2, R=2    # common default: survives 1 node down for both reads and writes
N=3, W=1, R=1    # fast, but a read can miss the latest write entirely
N=3, W=3, R=1    # fast reads, writes block if any replica is down

The overlap guarantees at least one node in your read set saw the newest write. There is no failover event because there is no leader to fail — a node dying just means quorums assemble from the survivors, and repair mechanisms (read repair, anti-entropy) backfill it later. The costs: you have version conflicts to resolve (vector clocks or LWW again), and "sloppy quorums" with hinted handoff — accepting writes on stand-in nodes during a partition — quietly weaken the overlap guarantee that the arithmetic promised. This is the CAP trade-off showing up in the fine print.

How to choose

Start from the question "how much data can we lose, and how stale can reads be?" — not from the technology.

• Default OLTP workload: leader-follower, semi-sync, automated failover with fencing, lag-aware read routing.
• Read scaling: add followers, but classify every read path as leader-required or staleness-tolerant. The dashboard can lag; the balance check cannot.
• Multi-region writes: single home region per record before multi-leader; multi-leader before you hand-roll conflict resolution.
• Always-writable under partitions, conflict-tolerant data: leaderless quorums.

Replication is the part of the stack where a config default (async, W=1) quietly becomes your durability policy. Make it a decision instead.