Design an API gateway

Every request to a company’s public API, 60,000 a second at peak across 900 routes, must have its caller checked, its limit counted and its path matched before any of 120 services sees it. An API gateway is the single front door that does this: a fleet of identical, stateless reverse proxiesreverse proxy: A server that clients reach as if it were the site itself. It passes each request to one of the servers behind it, often ending TLS, routing by path, balancing load and caching on the way. More in “Forward proxies, reverse proxies and gateways” that run every request through the filter chain of the route it matches (TLS, authentication, rate limits and quotas, request validation, routing to a service, response shaping and logging), so that services do not each implement them.

Two things make it a design problem rather than a product to install. Every request pays for the chain, so the gateway must stay thin and fast; and the route table and its policies are configuration that changes many times a day, so a bad change must never reach every instance at once.

Requirements and estimate: the loss of a zone sizes the fleet

The company’s 120 services are used by its mobile apps, its website and about 2,000 partners. Today every team runs its own token checks and limits, and they disagree. Ask first:

Functional requirements: route by host, path, method and headers; check tokens and API keys; rate limits and quotas; request size limits and schema validation; per-route timeouts and retries; access logs and traces. Non-functional requirements: a few milliseconds of overhead; no outage when a zone or any dependency fails; configuration changes without restarts; and a bad change contained to a small share of the traffic.

The first estimate, to change and run again:

# The first estimate for an API gateway fleet. Every input is an assumption, written down so that you can change it
# and run the script again.
import math

PEAK = 60_000              # requests a second at peak, every API together
PER_INSTANCE = 4_000       # requests a second one instance handles at its target CPU (TLS, filters, proxying)
ZONES = 3                  # instances spread evenly over 3 zones; the fleet must survive the loss of one
JWT_VERIFY_MS = 0.1        # CPU time to check one token's signature
CACHE_HIT = 0.9            # share of requests whose token this instance verified a moment ago
SYNC_MS = 100              # each instance reports its rate-limit counts this often, in one batch
LOG_BYTES = 1_000          # one access-log record
ROUTES, ROUTE_BYTES = 900, 2_000

load = math.ceil(PEAK / PER_INSTANCE)
zone_safe = math.ceil(load / ((ZONES - 1) / ZONES))
fleet = math.ceil(zone_safe / ZONES) * ZONES
print("Instances")
print(f"  for the load: {PEAK:,} / {PER_INSTANCE:,} = {load}")
print(f"  to survive losing 1 of {ZONES} zones: {load} / ({ZONES - 1}/{ZONES}) = {load * ZONES / (ZONES - 1):.1f}, "
      f"rounded up to {zone_safe}, and to {fleet} for the same number in every zone ({fleet // ZONES})")

print("Rate-limit decisions")
print(f"  a central counter asked on every request: {PEAK:,} calls a second, each a network round trip")
print(f"  local token buckets, counts sent every {SYNC_MS} ms in one batch: {fleet} x {1000 // SYNC_MS} "
      f"= {fleet * 1000 // SYNC_MS} messages a second")

print("Token checks")
print(f"  every request verified: {PEAK * JWT_VERIFY_MS / 1000:.1f} CPU cores")
print(f"  with {CACHE_HIT:.0%} of tokens verified a moment ago: {PEAK * (1 - CACHE_HIT) * JWT_VERIFY_MS / 1000:.1f} cores")

print("Logs and configuration")
print(f"  access logs: {PEAK * LOG_BYTES / 1e6:.0f} MB a second, {PEAK * LOG_BYTES * 86400 / 1e12:.2f} TB a day")
print(f"  route table: {ROUTES} routes x {ROUTE_BYTES:,} bytes = {ROUTES * ROUTE_BYTES / 1e6:.1f} MB per version, "
      f"sent to {fleet} instances")

Instances
  for the load: 60,000 / 4,000 = 15
  to survive losing 1 of 3 zones: 15 / (2/3) = 22.5, rounded up to 23, and to 24 for the same number in every zone (8)
Rate-limit decisions
  a central counter asked on every request: 60,000 calls a second, each a network round trip
  local token buckets, counts sent every 100 ms in one batch: 24 x 10 = 240 messages a second
Token checks
  every request verified: 6.0 CPU cores
  with 90% of tokens verified a moment ago: 0.6 cores
Logs and configuration
  access logs: 60 MB a second, 5.18 TB a day
  route table: 900 routes x 2,000 bytes = 1.8 MB per version, sent to 24 instances

Three numbers decide the design. The fleet is sized for the loss of a zone, not for the load. A central counter on every request is 60,000 calls a second, while batched local counts are 240 messages. And caching recently verified tokens saves nine signature checks in ten.

The first design: a filter chain per route, cheapest refusals first

Each part of the design has one job:

• Route table: maps host, path, method and headers to a route, with its backend, timeouts and filters.
• Filter chain: the ordered steps a route’s requests pass through, cheapest refusals first.
• Authentication filter: checks a token’s signature and claims, or an API key, before anything uses the identity.
• Rate limits and quotas: token bucketsA rate limiter that holds up to a set number of tokens and refills them at the allowed rate: each request takes one token, and a request that finds the bucket empty is refused or made to wait. per key and route, enforced locally and shared through a limit service.
• Request validation: size limits and the route’s schema, so malformed requests never reach a service.
• Control plane: stores routes and policies as versioned snapshots, validates them and rolls them out in stages.
• Backend for frontend: a client team’s own service for aggregation and shaping, outside the shared gateway.

Follow one partner request through its route’s filters:

The deep dives that decide whether the design works:

1. How is a request matched to a route, quickly and without being fooled by its path?
2. In what order do the filters run, and what does each do when its dependency fails?
3. How do limits stay accurate across 24 instances without a call on every request?
4. How does a configuration change reach the fleet without putting all of it at risk?
5. How do the issuer’s signing keys rotate without an outage?
6. What stays out of the gateway?

An answer is judged on requirements and estimates, routing, the filter chain, limits, configuration safety, failure handling and operations.

The API, and route tables swapped whole Pro: opens with a Pro, Premium or Ultimate pass