Designing Kafka Itself: The Distributed Log, ISR Replication, and Compaction

"Design a message queue" used to mean in-memory queues and consumer acks. Then Kafka reframed the problem: what if the queue is just a replicated append-only file that consumers read at their own pace? Most candidates can use Kafka; the interview that matters asks you to build it. Working through that design teaches you why the log — not the queue — turned out to be the right primitive for data infrastructure.

The core bet: sequential I/O and dumb brokers

A queue that tracks per-message delivery state (delivered? acked? redelivery timer?) holds mutable state per message per consumer — that state machine is the bottleneck. Kafka's move: the broker stores an immutable, append-only log per partition, and consumers own their position (an offset — just an integer). Delivery state for a million consumers is a million integers, stored by the consumers.

This buys three things at once: writes are sequential appends (disks, even spinning ones, love this — hundreds of MB/s); reads are sequential scans served straight from the OS page cache; and replay is free — rewind your offset. Add zero-copy (sendfile() from page cache to socket, bytes never enter user space) and one broker saturates the NIC without breaking a sweat.

partition = [seg0][seg1][seg2 (active)]
             each segment: log file + sparse offset index
consumer A offset: 41,022     consumer B offset: 12,904,551

Segments (say 1GB each) exist so retention is rm old_segment — deleting from the middle of a file is exactly the random I/O the design refuses to do.

Partitioning: the scaling and ordering contract

A topic is N partitions; each partition is an independent totally-ordered log living on some broker. Ordering is guaranteed within a partition only, so the partition key is a product decision: key by user_id and one user's events are ordered; hot users create hot partitions — the same skew trade every sharded system makes. Throughput scales by adding partitions; too many partitions bloats metadata, recovery time, and open file handles. (Modern Kafka moved cluster metadata from ZooKeeper to an internal Raft quorum — KRaft — precisely because partition counts kept growing.)

Replication: ISR and the high-watermark

Each partition has a leader and followers. Kafka's replication design is the part worth whiteboarding — it is neither Raft nor primary-backup, and its vocabulary shows up in incident reviews:

• Followers pull from the leader. Followers that are caught up (within a lag bound) form the ISR — in-sync replica set.
• A write with acks=all is acknowledged when every ISR member has it.
• The high-watermark = the last offset replicated to the whole ISR. Consumers can only read up to it — you never read a message that could vanish in a failover.
• Leader dies → controller elects a new leader from the ISR → nothing acked is lost.

The subtle knob: if replicas fall behind, they are ejected from the ISR, and the ISR can shrink to just the leader — acks=all then means "one machine has it." min.insync.replicas=2 is the guardrail: below two in-sync copies, the partition refuses writes. That is CAP played out in one config key — refuse availability rather than silently weaken durability. The pathological alternative, unclean.leader.election (let a stale replica lead), trades data loss for availability and is off in any system that bills anyone.

leader log:   m1 m2 m3 m4 m5
follower1:    m1 m2 m3 m4        (in ISR)
follower2:    m1 m2              (lagging -> ejected)
high-watermark = m4  -> consumers see up to m4

Exactly-once, the honest version

Producer retries duplicate messages; the fix is an idempotent producer — a producer ID plus per-partition sequence numbers lets the broker drop replays. Transactions extend this: writes to multiple partitions plus the consumer's offset commit become atomic (offsets are themselves a compacted internal topic — the design eats its own primitive). What transactions do not cover is a side effect into an external system; there you are back to idempotent sinks. Same lesson as every exactly-once story: it is exactly-once within the log's transaction boundary, at-least-once plus idempotency beyond it.

Log compaction: the log as a table

Retention has two modes. Time/size retention drops old segments — fine for event streams. Compaction instead keeps the latest value per key, scrubbing older duplicates in the background. A compacted topic is a materialized changelog: replay it and you rebuild a table (this is how Kafka Streams state stores and the consumer-offsets topic recover). Deletes are tombstones — (key, null) — retained long enough for consumers to observe, then scrubbed; if that sounds like Dynamo tombstones and CRDT deletes, it is — deletion in distributed systems is always a message, never an absence.

What the interviewer is actually probing

Question	The design answer
Why is it so fast?	Sequential appends + page cache + zero-copy; the broker does no per-message bookkeeping
Message vanished after failover?	Read past high-watermark can't happen; check ISR shrink + acks config
Consumer group rebalancing storms?	Offsets external to broker make consumers restartable; sticky/cooperative assignment limits churn
Queue vs log, when?	Per-message routing/priority/TTL → queue (SQS/Rabbit); replay, ordering, fan-out to many readers, stream processing → log

The transferable idea is bigger than Kafka: state your storage as an ordered immutable log and let every other concern — replication, delivery, recovery, even tables — become a reader position or a fold over it. Half of modern data infrastructure is that sentence wearing different logos.