Designing an S3-Style Object Store: Erasure Coding and the Metadata Layer

S3 promises eleven nines of durability — lose one object in ten million, on average, per ten thousand years. No single disk, machine, or even datacenter can make that promise; the design has to manufacture durability out of statistically unreliable parts. Designing an object store is really designing three loosely coupled systems — a metadata plane, a data plane, and a repair plane — and knowing why they must be separate.

Requirements and the API contract

PUT/GET/DELETE/LIST over objects (bytes + key + metadata) in buckets. Objects are immutable — you overwrite whole objects, never edit bytes in place — which is the simplification that makes everything else tractable. Durability 99.999999999%, availability ~4 nines, objects from 1KB to 5TB, strong read-after-write consistency (S3 has promised this since 2020).

Split the planes

Metadata plane: key → object version → list of chunk locations, plus buckets, ACLs, multipart state. Small records, strong consistency, high QPS. A sharded, replicated KV store (Raft groups per shard, or Spanner-style) — metadata is where consistency lives.

Data plane: dumb storage nodes holding immutable chunks. Operations: write chunk, read chunk, checksum chunk. No consensus in the hot path — the chunk placement decision was already recorded in metadata.

PUT /bucket/key
 1. API layer streams bytes, splits into chunks, checksums each
 2. picks placement (N nodes across failure domains), writes chunks
 3. commits metadata row {key, version, chunk_map} -- the atomic point
 4. ack client

GET = metadata lookup -> parallel chunk fetch -> verify checksums -> stream

The commit point is the metadata write. Chunks written but never committed are garbage (collected later); a committed row whose chunks are missing is a durability bug. Order of operations is the whole correctness argument: data first, then metadata, ack last. Read-after-write consistency falls out of the metadata store's consistency — this is why S3's move to strong consistency was a metadata-layer project, not a data-layer one.

Durability math: replication vs erasure coding

3x replication tolerates 2 failures at 3x storage cost. At exabyte scale, 3x is billions of dollars, so real object stores use erasure coding: split an object's data into k data fragments, compute m parity fragments (Reed-Solomon), store the k+m fragments on different machines/racks/AZs. Any k of the k+m fragments reconstruct the data.

Scheme	Overhead	Tolerates	Read path
3x replication	3.0x	2 losses	1 node
RS(6+3)	1.5x	3 losses	6 nodes
RS(12+4)	1.33x	4 losses	12 nodes

EC's fine print, which is where the interview signal is: reads touch k nodes (worse tail latency — a straggler among 12 hurts more than among 1); small objects fit poorly (fragment overhead dominates — so small objects are 3x-replicated or packed into larger blocks and EC'd together); and repair traffic is enormous — rebuilding one lost fragment reads k fragments' worth of data. A common hybrid: replicate on ingest (fast, simple), erasure-code in the background once the object cools.

Placement must respect failure domains: fragments spread across racks and availability zones so correlated failures (switch, power, flood) do not take out more than m fragments. Placement is a constraint solver run at write time; its output is just rows in metadata.

Repair: durability is a verb

Eleven nines does not come from writing fragments once. Disks fail at ~1-2% AFR and, worse, rot silently (latent sector errors, bit flips). Durability is maintained by an always-on background plane:

• Scrubbing: continuously read every chunk, verify checksums, flag corruption. Every byte gets re-verified on a cycle measured in weeks.
• Repair queue: node/disk death enqueues re-replication or fragment reconstruction; the fleet burns a sizable, budgeted fraction of its I/O on repair at all times.
• The durability model is a race: MTTR vs the probability of m+1 overlapping failures. Halving repair time buys more durability than adding a parity fragment — which is why repair bandwidth is a first-class capacity-planning line item, not best-effort.

The parts everyone forgets

LIST is a range scan over the key space (prefix/... ordering), which is why metadata wants an ordered store (or an ordered index beside a hash store) — and why hot sequential key prefixes (logs/2026-07-13-*) can hammer one metadata shard. S3's per-prefix throughput guidance is that shard boundary showing through.

Multipart upload is metadata choreography: initiate → upload parts independently (each its own chunk set) → complete assembles the part list into one object version atomically. It exists because a 5TB single-stream PUT over a flaky link must be resumable per-part.

DELETE is a tombstone in metadata; chunks die later via garbage collection reconciling "chunks on disk" vs "chunks referenced" — with versioning, lifecycle rules, and in-flight multiparts as edge cases. GC bugs are how object stores actually lose data; deleting too eagerly is the catastrophic direction, so GC runs conservative, delayed, and audited.

The summary shape: consistency lives in a small metadata plane; scale lives in a dumb immutable data plane; durability lives in a repair plane that never sleeps. If your design has those three boxes and the commit point in the right place, the rest is parameter tuning.