Designing a Distributed Lock Service: ZooKeeper and Chubby from the Inside

Ask a room of engineers how their systems elect leaders or claim shards and the answer is "ZooKeeper" or "etcd" — infrastructure so standard it's invisible. Ask them to design it and the room quiets, because the lock service sits at the bottom of the trust stack: everything above assumes it's correct, so it gets no layer beneath it to lean on. Google's Chubby paper (which ZooKeeper reimplemented in spirit) is the design, and it's full of decisions that only make sense once you've asked "what do coordination clients actually need?"

What it is: a tiny, replicated, watchable filesystem

The core is a consensus-replicated state machine (Paxos in Chubby, ZAB in ZooKeeper, Raft in etcd — five replicas, leader-based, majority commit) exposing not lock()/unlock() RPCs but a small hierarchical namespace of nodes (znodes) with data, versions, ACLs — plus three primitives that turn a filesystem into a coordination kernel:

• Sessions with leases: a client's connection is a lease kept alive by heartbeats; the service and client agree on its expiry.
• Ephemeral nodes: znodes bound to a session — session dies, nodes vanish. This is the failure detector, packaged as data: "the lock holder crashed" becomes "the lock file disappeared."
• Watches: one-shot notifications on node changes — "tell me when this file changes/vanishes" — so coordination is event-driven, not polled.

Locks, elections, membership, and config all compose from these instead of being API surface: a lock is an ephemeral node you managed to create; a leader is whoever holds /election's lowest sequence node; group membership is the children of /members (each an ephemeral node); config is a watched file. Primitives over verbs — the design's first big lesson.

The recipes, and their sharp edges

Lock without a herd: naive locking ("everyone tries to create /lock; losers watch it") wakes every waiter on release — a thundering herd per unlock. The standard recipe uses sequence nodes:

acquire: create /lock/waiter-  (ephemeral + sequential -> waiter-0007)
         list children; am I lowest? -> hold the lock
         else: watch ONLY my immediate predecessor (waiter-0006)
release: delete my node (or die -> ephemeral cleanup does it)
         -> exactly one waiter (0008) wakes

One watch per waiter, one wake per release, FIFO fairness for free. This queue-of-ephemera pattern is leader election with a different node name.

The lock that lies: the deepest sharp edge, and the interview's real test. A client holds the lock, GC-pauses for 40 seconds, its session expires, the lock passes on — and then the original client wakes up and keeps writing, convinced it holds the lock. The lock service cannot prevent this (it can't reach into a paused process); it can only give you the tool: a fencing token — the lock node's monotonically increasing version/zxid — that the client must present to downstream resources, which reject stale tokens. A lock service without fenced consumers is a mutual-suggestion service; Chubby's sequencers exist precisely because Google learned this the hard way.

Scaling reads without breaking the promise

The service is read-dominated (configs and membership get read constantly, changed rarely), so both systems lean hard on caching/local reads: Chubby clients hold consistent caches invalidated synchronously by the master before a write completes (writes pay latency so reads can be local and never stale); ZooKeeper lets any replica serve reads (fast, possibly stale) and orders everything by zxid, offering sync for read-your-writes when it matters. Same fork as always — pay at write time or acknowledge staleness at read time — chosen differently by two systems with the same job. Watches keep herds down (invalidation, not data push; clients re-read on notification), and session events (DISCONNECTED) force clients to assume the worst about their own locks during partitions — the client library's state machine is half the system's correctness.

Chubby's paper is bluntly honest about the operational discovery: the service built for coarse locks became, overwhelmingly, a name service and config store — because "small consistent files + watches + ephemerality" is what everyone actually needed. Coarse-grained locks (held for hours: leadership, shard ownership) fit the design; fine-grained locks (per-request) never should touch it — a consensus round per acquisition would melt it, which is why per-key locks live in application stores (the Redis/Percolator territory) with the lock service anchoring only the epoch those schemes fence against.

Interview probe	Answer sketch
Why five replicas, one leader?	Majority quorum survives two failures; a leader gives writes a serialization point; reads scale out — the standard consensus deployment shape
Session timeout tuning?	The liveness/safety dial: short = fast failover + spurious expiry under GC/network blips; long = slow failover; Chubby's answer includes client-side grace periods and jeopardy warnings
What if the lock service itself partitions?	Minority side serves no writes (and Chubby caches go conservative); clients see session jeopardy and must stop acting on held locks — unavailability is the correct behavior
etcd vs ZooKeeper vs Consul?	Same kernel (consensus + lease + watch + KV); differences are API ergonomics, watch semantics, and ecosystem — say the kernel, don't relitigate logos

The distilled design: consensus-replicated small state, leases as failure detection, ephemerality as cleanup, watches as invalidation, sequence numbers as fairness and fencing. Every coordination pattern you've ever used is five lines of recipe over those five properties — and the fencing token is the line most systems forget until an incident writes it for them.