Designing a WebSocket Gateway: 100 Million Persistent Connections

Stateless HTTP scaling has one rule: any server can handle any request. WebSockets repeal it. A connection lives on exactly one machine for minutes or hours, holds kernel and application state there, and everything that wants to reach that user must find that machine. Every chat app, live dashboard, multiplayer game, and notification system contains this system — a connection gateway — and it has almost nothing in common with the stateless tier behind it. Designing it well is a distinct skill, and interviewers increasingly carve it out as its own question.

Requirements

Hold 100M concurrent connections; deliver a message to any connected user in under 100ms; survive node failures and — harder — routine deploys; handle reconnect storms after regional blips; auth, heartbeats, and per-connection backpressure.

Per-node limits: the real ones

The mythical 65K-port ceiling is irrelevant (that limit applies per client source-IP tuple; a server accepts millions on one port). Actual per-node budgets: memory (~tens of KB per idle connection across kernel buffers + TLS session + app object — 1M connections ≈ 30-50GB before message buffers), file descriptors (a config line), and event-loop discipline (epoll/kqueue-based runtimes — Netty, Go, Erlang — not thread-per-connection). Practical dense nodes run 500K-1M connections; 100M users therefore means 100-200 gateway nodes, and the interesting problems are all fleet-level.

The routing problem: who holds user X?

A message for user X must reach the node holding X's socket. Two workable shapes:

Connection registry (the common one): on connect, gateway writes user → node to a shared store (Redis/etcd) with a TTL refreshed by heartbeat; senders look up and RPC the owning node directly.

connect: user X -> LB -> gateway-17; registry[X] = {node:17, ts}
send:    lookup X -> gateway-17 -> push down socket
failover: node dies -> TTLs lapse -> clients reconnect elsewhere -> registry heals itself

Pub/sub fabric (the decoupled one): every gateway subscribes to topics for its connected users (user.X); senders publish blindly. No registry to keep correct — the broker's subscription table is the registry — at the cost of a broker tier sized for your entire message volume. Hybrid is common: registry for point-to-point, pub/sub for broadcast/rooms.

Multi-device is free in either shape (registry holds a set; or both devices' gateways subscribe), and it forces the semantic question worth raising unprompted: delivery to user means fan-out to all their sockets, dedupe client-side by message ID.

The two storms

Deploys. Restarting a gateway disconnects a million users — so a naive rolling deploy of 150 nodes disconnects everyone, hourly if you ship hourly. The playbook: drain mode (stop accepting new connections; existing ones get a RECONNECT_SOON frame with a jittered deadline), clients reconnect gracefully elsewhere at their leisure, node exits when empty or at timeout. Long-lived connections turn "deploy" from an instant into a process — budget 10-15 minutes per batch and design the fleet math around perpetual partial drain.

Reconnect stampedes. A regional blip drops 20M connections; all 20M phones retry now — plus TLS handshakes (the expensive part) plus auth checks plus registry writes, a self-inflicted DDoS arriving exactly at recovery. Defenses in order of leverage: client-side exponential backoff with jitter baked into the SDK (the single highest-value line of code in this system), server admission control (accept-rate limiting; shed excess with Retry-After), TLS session resumption to cheapen handshakes, and auth-token validation that skips the identity service on reconnect (short-lived signed tokens verified locally).

Semantics: the connection is not the delivery

The socket delivers at-most-once: anything in flight when a connection drops is gone. Real delivery guarantees live one layer up — per-user sequence numbers and a short replay buffer (or a durable per-user queue for messaging products): client reconnects with last_seq=1041, gateway (or the message service behind it) replays the gap. This split — dumb pipe, smart resume protocol — keeps gateways stateless-enough to drain and crash freely; the resumable state lives in a store sized for undelivered messages, not all traffic.

Per-connection backpressure closes the loop: a phone on 2G can't absorb a 10K msg/sec room. Bounded per-socket send queues; on overflow, policy by product — drop-oldest with a "you're behind" marker (dashboards, tickers), conflate to latest-state (presence, cursors — the leaderboard trick again), or disconnect-and-resume (messaging). Unbounded queues here are how one slow client OOMs a node holding a million healthy ones.

Heartbeats run at two levels — TCP keepalive is insufficient (NAT/LB idle timeouts, typically 60-350s, kill silent connections; and a dead peer can take kernel-minutes to surface) — so the protocol pings every ~30s, which doubles as the registry TTL refresh and the liveness signal for "deliver vs queue" decisions.

Interview probe	Answer sketch
Why not sticky sessions on the LB?	Stickiness solves routing in, not reaching the user from inside — the registry/pub-sub problem remains; and stickiness fights draining
Registry entry stale (node died with TTL unexpired)?	Send RPC fails fast → treat as offline → queue + push notification path; TTLs bound the lie to seconds
One user in 10M-subscriber room?	Rooms are pub/sub topics; per-gateway single delivery + local fan-out to that node's members — never per-user publishes
WebSocket vs SSE vs long-poll?	Bidirectional + binary + one FD → WS; server-push-only telemetry → SSE is operationally simpler (plain HTTP); long-poll is the fallback, not the design

The keep-this shape: a fleet of dense, drainable socket-holders; a registry or broker answering "who holds X"; sequence-numbered resume one layer up; and jittered backoff in every client you ship. Get those four right and the other hundred details are tuning.