Load Balancing Deep Dive: L4 vs L7, Power of Two Choices, and Subsetting

Load balancing gets one slide in most system design answers — a box labeled "LB" that sprays requests evenly. In production, that box is a stack of decisions that determine your tail latency more than most application code does: which layer balances, with what information, using which algorithm, across which subset of backends. The gap between "round-robin at the LB" and what Google/Meta-scale serving actually does is a full senior interview's worth of material.

The layers: L4 spreads connections, L7 spreads requests

L4 (transport) balancers route connections by 5-tuple hashing — no request parsing, millions of connections per node, often implemented as anycast VIPs + ECMP + consistent hashing (Maglev/Katran-style, in kernel or XDP). Cheap and blind: with HTTP/2 multiplexing many requests over few connections, per-connection balance can hide brutal per-request imbalance — two connections can carry wildly different loads to their pinned backends.

L7 (application) balancers parse requests and balance each one: per-request routing (path/header rules), retries, and — critically — per-request load awareness. The standard deployment is both: L4 at the edge to spread connections across an L7 fleet (the gateway/sidecar proxies), L7 making the real balancing decision per request. Saying "HTTP/2 broke L4-only balancing" and why is a strong early signal.

Algorithms: why P2C beats the clever ones

Round-robin ignores that requests and backends aren't uniform. Least-connections sounds smarter but has a classic pathology in distributed balancers: many independent LB nodes each pick their own least-loaded backend using slightly stale stats — and they all pick the same one, dogpiling it into a spike, then stampeding to the next (the "thundering herd of the informed"). Stale global knowledge synchronizes mistakes.

Power of two choices (P2C): pick two backends at random, send to the less loaded of the pair.

p2c: a, b = random(backends, 2)
     send to argmin(load(a), load(b))    # load = outstanding requests / EWMA latency

The randomness desynchronizes the balancers, so stale data can't cause dogpiles, while the "of two" comparison still steers sharply away from overload — the max-load bound improves exponentially over pure random with just that one extra sample. It's the default in Envoy and most modern meshes for exactly this reason: nearly the quality of global-knowledge balancing with none of its herd behavior. Load signal choice matters too: outstanding-requests (cheap, local) or latency-EWMA (catches degraded-but-accepting backends); both are local observations, which is the point.

Two refinements every large fleet adds: slow start (new/restarted backends ramp weight over ~30s — a cold JIT/cache instance receiving full share instantly is how deploys create latency spikes), and outlier ejection (a backend with anomalous error/latency gets temporarily removed — the circuit breaker as an LB-native feature, per-host).

Subsetting: when everyone can't talk to everyone

With 10K clients and 10K backends, full-mesh connection counts (10⁸) drown memory and health-check traffic. Deterministic subsetting gives each client a small subset (say 20-50 backends) chosen so that: subsets overlap minimally, every backend appears in ~equal numbers of subsets (else you've built imbalance), and churn (backend added/removed) reshuffles few assignments. Random-but-seeded shuffles per client group (the Google SRE-book construction) achieve this; naive "hash client to K backends" doesn't (uneven backend coverage). Subsetting interacts with P2C fine — you pick two within your subset — and with locality: zone-aware weighting keeps most traffic intra-zone for latency and egress cost, spilling cross-zone proportionally as local capacity degrades.

For stateful backends (caches especially), balancing flips to consistent-hash/ring routing — the LB's job becomes affinity (same key, same node) rather than evenness, with bounded-load consistent hashing capping how hot any node may get before spilling. Choosing "affinity vs evenness" per service is the design decision; there is no universal algorithm.

Health, and the herds hiding in it

Health checking is its own distributed systems problem: active probes (per-LB × per-backend — subsetting saves you here too), passive outlier detection from live traffic, and the panic threshold — when >X% of backends look unhealthy, stop trusting the health data and route to everything, because a correlated "everyone unhealthy" signal usually means the checker or a shared dependency is lying, and routing all traffic to the three "healthy" survivors kills them instantly (the load-shedding cousin of Lifeguard's "maybe I'm the problem"). Connection draining on deploys (finish in-flight, refuse new, then die) and retry budgets at the LB tier round out the failure choreography — the L7 balancer is where retries amplify or get contained.

Interview probe	Answer sketch
Least-conn vs P2C, one line?	Global "best" + stale data = synchronized dogpiles; random two + local compare = desynchronized near-best — pick P2C
gRPC (long-lived HTTP/2) balancing?	Client-side L7 balancing via resolver/mesh (per-RPC picks), or L7 proxies; L4 alone pins whole channels to hosts
One backend at 90% CPU, others idle?	Check: hash affinity hotspot? subset imbalance? slow-start missing after deploy? — the diagnosis tree is the answer
DNS as the balancer?	TTL-bound staleness, no load signal, client disobedience — fine for coarse geo-steering (the CDN's job), never for backend evenness

The distilled hierarchy: L4 spreads connections, L7 spreads requests; P2C with local load signals beats stale global cleverness; subsetting bounds the mesh; slow-start and panic thresholds tame the herds. Load balancing done well is invisible — which is exactly why it's worth being the candidate who can make it visible on a whiteboard.