Raft in Plain English: How Clusters Agree When Nodes Keep Dying

Somewhere under every reliable distributed system sits a small cluster of three or five nodes whose only job is to agree on things: who the leader is, what the config is, which worker owns which partition. Kubernetes trusts etcd. Consul and Nomad run it internally. Kafka replaced ZooKeeper with KRaft. CockroachDB runs it per range. All of them are running Raft, and if you understand Raft's two moving parts — elections and log replication — the behavior of all of these systems under failure stops being mysterious.

The problem Raft solves: get a cluster of unreliable machines to maintain one agreed-upon, ordered log of operations, such that once an entry is committed, no crash, restart, or network partition can un-commit it. An ordered log is enough to build anything — apply the same operations in the same order to a state machine on every node, and every node computes identical state. That is the replicated state machine pattern, and Raft's whole job is protecting the log.

Terms and elections

Raft time is divided into terms, numbered forever upward. Each term has at most one leader. The term number is Raft's logical clock: every message carries it, and any node that sees a higher term than its own immediately steps down to follower and adopts it. Stale leaders discover their staleness the first time they talk to anyone.

Every node is a follower, candidate, or leader. Followers expect periodic heartbeats from the leader. When a follower hears nothing for an election timeout (randomized, say 150–300ms), it assumes the leader is dead:

1. Increment term, become candidate, vote for self
2. Request votes from all peers
3. Peers grant one vote per term, first-come-first-served
   (with a safety check — see below)
4. Majority of votes → leader. Start sending heartbeats.
   Someone else's heartbeat with equal/higher term → step down.
   Timeout with no winner → new term, try again.

The randomized timeout is doing quiet, load-bearing work: it makes simultaneous candidacies (split votes) rare, and after a genuine leader death, whichever node times out first usually wins cleanly. Elections settle in tens to hundreds of milliseconds — which is also your write-unavailability window during a failover.

Majorities are why cluster sizes are odd. Three nodes tolerate one failure; five tolerate two. Four nodes still need three votes, so they also tolerate only one failure — you added hardware and gained nothing but a bigger quorum to wait on.

Log replication

Once elected, the leader accepts client writes and drives them through the cluster:

1. Leader appends entry to its local log: (term=7, index=42, op)
2. Leader sends AppendEntries to all followers
3. Each follower appends and acknowledges
4. Once a majority (leader included) has the entry,
   the leader marks it COMMITTED, applies it, responds to client
5. Followers learn the commit point in subsequent messages and apply too

Committed means "on a majority of disks," not "on all." Slow or dead followers never block the cluster — the majority moves on, and stragglers catch up later: AppendEntries carries a consistency check (previous entry's index and term must match), and on mismatch the leader backs up and overwrites the follower's divergent tail until the logs agree. Uncommitted entries on a minority can be discarded by history; committed entries never are.

The one safety trick worth internalizing

Why can a committed entry never be lost? Two overlapping-majority arguments:

1. A committed entry lives on a majority of nodes.
2. Winning an election requires votes from a majority of nodes.
3. Any two majorities share at least one node — so every possible winner's electorate contains someone who has the committed entry.
4. The safety check mentioned above: a node refuses to vote for a candidate whose log is less up-to-date than its own.

Therefore a candidate missing committed entries cannot assemble a majority — the overlap voter rejects it. The election mechanism itself guarantees the new leader already has everything that was ever committed. No log transfer, no reconciliation protocol at failover; the vote is the reconciliation. This is the cleverest ten lines in the paper, and it is also the answer to the interview follow-up "how does the new leader learn the old leader's committed writes?" — it never has to.

Split brain falls out too: a partitioned old leader keeps a minority, cannot commit anything (no majority acks), and steps down the moment it hears the new term. Clients that wrote to it never got success responses for those entries.

What Raft costs and where it fits

Every write costs a round-trip to a quorum and a disk fsync on each — consensus is expensive. Throughput is bounded by a single leader; latency is bounded by the median follower. Reads default to going through the leader too (a lease-based optimization lets leaders serve reads locally while their lease holds).

That is why nobody runs their data plane through a single Raft group. The pattern is a small consensus core holding the coordination state — cluster membership, config, leases, shard ownership — while the data path scales out separately: replication for the data itself, with Raft deciding who the primary is. Systems like CockroachDB run many Raft groups, one per data range, so no single group is a bottleneck.

Also worth knowing what Raft does not give you: exactly-once client semantics (a retried command can commit twice without idempotent handling), Byzantine fault tolerance (nodes are assumed honest, just crash-prone), and cross-group transactions.

Takeaways

Raft is elections plus a replicated log, glued by term numbers and the rule that majorities overlap. Committed means majority-persisted, and the up-to-date voting restriction makes leader changes lossless by construction. Size clusters odd, expect a sub-second write freeze during elections, and keep consensus in the coordination plane where its cost buys correctness — not in the hot path where it buys latency. Understand these mechanics once, and etcd's behavior during a node failure, Kafka's controller failover, and your next system design interview all become the same conversation.