Lakehouse Table Formats: How Iceberg and Delta Put ACID on Object Storage

The old data lake was a directory convention: a "table" meant "all the Parquet files under this path," discovered by listing. That convention collapses under real use — a job writing files mid-query makes readers see half a commit; renaming a partition directory is a multi-minute non-atomic disaster; and LIST on a million-file prefix costs more than the query. Table formats (Iceberg, Delta Lake, Hudi) fix this with one move worth internalizing: a table is not a directory — it's a metadata tree whose root is swapped atomically. That's how ACID lands on storage that offers nothing but immutable blobs and one conditional write.

The metadata tree

Iceberg's layout (Delta's log-based variant differs in mechanics, same effect):

table metadata (root)  ->  snapshot list (every committed version)
snapshot               ->  manifest list (one per snapshot)
manifest list          ->  manifests (each covering a set of data files,
                            with per-manifest partition ranges)
manifest               ->  data files + per-file stats
                            (row count, per-column min/max, null counts)

Data files are immutable Parquet; every write creates new files and new metadata describing the complete new table state. A commit is a single compare-and-swap of the root pointer — via the catalog (a tiny transactional store: Glue, a database row, or now S3 conditional PUTs) that says "current root = X." Readers resolve the root once and then read a frozen, fully consistent snapshot; writers who lose the CAS race retry against the new root (re-validating conflicts). Optimistic concurrency, isolation via immutability, atomicity via pointer swap — the whole ACID story in three clauses, built from parts S3 actually has. (If this rhymes with the S3-object-store post's "commit point is the metadata write" and the LSM post's immutable files — that's the point; it's the same design one level up.)

Why queries got fast: stats instead of listing

The metadata tree replaces LIST entirely — the manifests are the file inventory — and every layer carries pruning stats. A query with WHERE region = 'EU' AND day = '2026-07-14' skips whole manifests by partition range, then skips individual files by column min/max, often touching 0.1% of the table without a single storage listing:

planning: root -> snapshots -> prune manifests by partition ranges
          -> prune files by min/max stats -> read surviving Parquet

Iceberg's hidden partitioning fixes the classic Hive foot-gun: the table stores the transform (day(ts), bucket(user_id, 64)) as metadata, queries filter on the raw column, and the planner applies the transform — so analysts can't full-scan by forgetting the magic partition column, and partitioning can evolve (change granularity going forward) without rewriting history, because it's metadata, not directory structure.

Time travel, streaming, and the maintenance loop

Snapshots make time travel free (AS OF an older root) — reproducible ML training and audits fall out of retention policy. The snapshot diff gives incremental consumption — "files added since snapshot N" — which is CDC-shaped reading for downstream pipelines. Row-level changes (GDPR deletes, upserts) get two strategies, and choosing is the interview question: copy-on-write (rewrite affected files at write time — expensive writes, pristine reads) vs merge-on-read (write small delete/patch files, merge at query time — cheap writes, taxed reads). MoR for streaming ingest and rare hot updates, CoW for read-heavy analytics; Hudi built its identity on this dial, Iceberg and Delta now offer both.

Immutability's bill is a background maintenance plane, exactly like an LSM's: compaction (thousand tiny streaming-ingest files → fewer big ones; small-files is the operational disease of lakehouses), snapshot expiry (drop old roots, then garbage-collect unreferenced files — the delete-too-eagerly direction is the catastrophic one, so GC is conservative and audited), and stats/clustering maintenance (sort/z-order rewrites to make min-max pruning actually prune). A lakehouse without scheduled maintenance degrades into the directory soup it replaced.

Interview probe	Answer sketch
Two writers commit simultaneously?	Both write files; one wins the root CAS; loser revalidates (disjoint files/partitions → auto-retry merges; true conflict → abort) — contention is per-table commit, so high-frequency writers batch
vs a data warehouse?	Same table semantics, open files + open metadata on your storage, many engines (Spark/Trino/Flink/DuckDB) sharing one copy — the trade is you now operate the maintenance plane a warehouse hides
Streaming upserts at 100K rows/sec?	MoR + frequent minor compaction; CoW would rewrite the world per micro-batch
Why did Hive tables fail?	Directory-as-truth: non-atomic renames, list-based planning, partition columns as landmines — every failure is "filesystem semantics where table semantics were needed"

The distilled idea: immutable data + a tiny mutable pointer = transactions on anything. Once the only mutation in the system is one CAS, every hard property — isolation, time travel, incremental reads, concurrent engines — becomes a property of metadata layout rather than storage heroics.