Designing YouTube: Upload, Transcoding Pipelines, and Adaptive Bitrate

A video platform is two systems wearing one product: a write path that is one of the largest batch-compute pipelines on earth (500 hours of video arrive every minute, each needing to become ~dozen encoded renditions), and a read path that is "just" static file serving — at a scale where it consumes double-digit percent of global internet traffic. The interview mistake is spending forty minutes on either half. The design skill is noticing they share almost nothing: different consistency needs, different scaling laws, different failure tolerance.

Requirements

Upload videos up to hours long; playable within minutes of upload; smooth playback across devices and network conditions; ~1B daily watch hours; a lost upload is unacceptable, a delayed one is fine.

Upload: resumable by contract

Multi-GB files over consumer connections will be interrupted, so upload is chunked and resumable (the same protocol shape as S3 multipart): client requests an upload session, sends chunks with offsets, finalizes. Chunks land directly in object storage via pre-signed URLs — video bytes should never transit your application servers. On finalize, the raw file is durable, a video row exists in state UPLOADED, and a message enters the processing queue. Everything after that is asynchronous — the user sees "processing," and the system owes them an eventual notification, not a response.

Transcoding: a DAG, not a job

The naive model — "run ffmpeg on the file" — fails the moment you ask how a 4-hour video becomes playable in minutes rather than after 8 hours of serial encoding. Production pipelines model processing as a DAG executed by a fleet:

probe -> split into segments (~2-10s at keyframes)
      -> fan out: encode(segment × rendition matrix)   <- the parallel middle
      -> fan in:  stitch manifests, generate thumbnails, captions,
                  content-ID fingerprint, moderation scan
      -> publish (metadata flips to READY, CDN warm begins)

Split the video into segments, encode segments in parallel across hundreds of workers, then assemble. A 4-hour video becomes ~2,000 independent 10-second encode tasks; wall-clock time collapses from hours to minutes, bounded by the slowest segment. The rendition matrix (say 8 resolutions × 2-3 codecs) multiplies tasks again — one upload can be 30,000 tasks, which is why the scheduler is the real system: priority queues (new upload from a 50M-subscriber channel ≠ decade-old video re-encode), retry-with-idempotency per task (encoding is pure — same input segment, same output — so at-least-once execution is free), spot/preemptible instances for the bulk tier, and backpressure to admission.

Codec economics drive a second scheduling dimension: popular videos earn expensive encodes. Everything gets fast H.264; predicted-popular content gets VP9/AV1 passes that cost multiples of CPU but save 30-50% bandwidth forever after. Compute spent once versus bandwidth paid per view — a cost model worth saying out loud in the interview.

ABR: the client is the adaptive part

Adaptive bitrate is widely misunderstood as a server feature. In HLS/DASH, the server is dumb: it hosts small immutable segment files plus a manifest listing renditions and segment URLs. The player measures its own throughput and buffer, and picks which rendition's next segment to fetch:

buffer low  + throughput dropping -> fetch 480p segment next
buffer full + throughput high    -> step up to 1080p

All intelligence client-side, all content cacheable static files — the design reason HTTP-based streaming beat stateful protocols (RTMP) completely. Segment boundaries align across renditions (same keyframe cadence) so switching is seamless. Live streaming is the same machinery with a rolling manifest and a short DVR window; latency comes from segment duration × buffer depth, which is why low-latency modes shrink segments to ~1s chunks and pay for it in CDN request volume.

Read path: CDN with a popularity cliff

Views follow a brutal power law: a sliver of videos serve nearly all traffic. That makes the CDN story tiered by construction — hyper-popular content pinned in edge memory (including ISP-embedded caches; Google's OpenConnect-equivalent boxes inside carrier networks), the long tail served from regional origins on cache miss, and the cold tail living only in origin storage with lower-cost encoding. Metadata (titles, counts, comments) is the usual read-heavy cached-database problem and intentionally boring; view counting at this scale is the ad-click-aggregation problem — approximate live counters reconciled by batch, and nobody should promise exactness on the view counter.

Interview probe	Answer sketch
Why is my video "processing"?	DAG mid-flight; publish flips atomically when the required rendition set completes — partial renditions can go live progressively (SD first, 4K later)
Encode worker dies mid-segment?	Idempotent pure task, re-queued; segment outputs content-addressed so duplicates collapse
First hour of a viral video?	Upload region encodes fast path; CDN fills on demand; predicted-hot content pre-warmed to edges
Storage math?	Raw + all renditions ≈ 2-3× raw; renditions are re-derivable from raw, so replicate raw hard, renditions cheaply

The shape to remember: an ingest pipeline that is a distributed build system, and a delivery tier that is a static-file CDN with a popularity cliff — joined only by object storage and a metadata flip.