Designing an API Gateway: The Front Door as a System

Every request to every service passes through the gateway, which makes it simultaneously the most leveraged place to put logic and the most dangerous place to put bugs. Its job description is contradictory on purpose: do authentication, routing, rate limiting, retries, transformations, and observability for the entire company — while adding about a millisecond and having no state worth losing. Designing one is really designing a discipline about what belongs at the edge, plus the data plane that executes it fast.

Data plane / control plane: the load-bearing split

The architecture that every serious gateway (Envoy being the reference) converged on:

CONTROL PLANE: config API, validation, canarying of config itself
   -> pushes desired state (routes, clusters, policies) via streaming API (xDS-style)
DATA PLANE: fleet of stateless proxies, hot-reload config in memory
   -> the request path; touches nothing but local state + async lookups

The discipline: the data plane never blocks on the control plane. Proxies run from an in-memory snapshot; control-plane outages mean "config freezes," never "traffic stops." Config updates apply atomically per-snapshot (no half-applied route tables), version-stamped, with the same canary treatment as code — because a bad route pushed everywhere at once is the modern "we took ourselves down" incident. If this rhymes with the experimentation platform's config distribution and the CDN's purge plane, it should — shipping validated state to a fleet in seconds is its own reusable subsystem.

The request path: a filter chain

Per request, an ordered pipeline of small stages — the plugin/filter chain (Envoy filters, Kong plugins):

TLS terminate -> decode -> [authn] -> [authz] -> [rate limit] -> [transform]
             -> route match -> [retry/timeout policy] -> upstream LB -> response filters

Design rules that keep the chain honest. Every filter has a latency budget and a failure policy — auth fails closed (obviously), rate limiting usually fails open with a local fallback (the rate-limiter post's whole dilemma, embedded here as one filter's config), header transforms never do I/O. Anything requiring a remote lookup is async and cached: JWT validation is local (verify signature against cached JWKS keys — rotate via control plane) versus opaque tokens requiring an auth-service call with aggressive caching and stampede coalescing; the difference is the reason JWTs won at the edge. Transformations stay shallow — header mapping, path rewriting, maybe protocol translation (HTTP↔gRPC); the moment the gateway parses and rewrites bodies per-route, it has become an unversioned application no one owns (the "smart pipes" ESB mistake, relearned each decade). Aggregation-shaped needs (one mobile call fanning to five services) go to an explicit BFF layer that is owned application code, not to gateway plugins.

Per-route resilience policy is gateway-owned because it must be uniform: timeouts (deadline propagation via headers so downstream stops working on abandoned requests), bounded retries with budgets (retry storms amplify outages; a global retry budget — max % of traffic that may be retries — is the guardrail), circuit breaking per upstream, and hedging only where idempotency is declared. The gateway is where "is this route idempotent?" becomes machine-readable metadata rather than tribal knowledge.

Multi-tenancy and the blast-radius problem

The gateway serves every team, so its failure domains need engineering: per-route/per-tenant resource isolation (connection pools, concurrency limits — one team's slow upstream must not eat the proxy's worker threads for everyone), config namespacing with per-team self-service (the control plane enforces schema + policy: teams own routes, platform owns global filters), and shard the fleet by criticality when scale allows — payments traffic on its own gateway pool means a bad plugin rollout to the general pool can't touch it. WAF/bot filters slot in as early filters; TLS and HTTP/2-3 termination happen once here so backends speak plaintext HTTP/1 or gRPC internally.

Observability is the quiet superpower: the gateway sees every request, so it's where golden signals per route, distributed-trace initiation (the traceparent header born here), and access logs come from uniformly — half the value of owning a gateway is that instrumentation becomes a platform property instead of a per-team chore.

Interview probe	Answer sketch
Gateway vs service mesh?	Same proxy tech, different plane: gateway = north-south edge concerns (client auth, tenancy, WAF); mesh sidecars = east-west (mTLS, service-to-service policy) — complementary, not rivals
Won't it become a bottleneck?	Stateless horizontal fleet; per-node ~50-100K RPS; the real risks are config blast radius and plugin bloat, not throughput
Where does rate limiting state live?	Local buckets + shared store for global limits — the two-tier limiter, deployed as a filter
One team needs request-body-based routing?	Push back: body-parsing at the edge taxes everyone; if truly needed, isolated route with explicit budget, or a BFF

The distilled design: a stateless filter-chain data plane fed by a versioned, canaried control plane; local-first decisions with cached async lookups; uniform resilience policy as config; and ruthless discipline about what does not belong at the edge. Gateways fail as products when they're slow, but they fail as platforms when they become the application.