Design a DNS-based global traffic manager

When a region fails, its users keep reaching it until the DNS answers cached for them expire: up to 104 seconds in the design below. A DNS-based global traffic manager is the authoritative DNS serviceauthoritative name server: A DNS server that holds a zone's records and answers for them itself, without asking any other server. Steering policies such as GeoDNS and health-checked failover run here. More in “DNS, GeoDNS and anycast for system design” for names such as api.example.com: it answers each query with the address of a healthy region near the user, so steering and failover happen in the answer itself.

It has four parts: anycastanycast: Announcing the same IP address from several sites, so that Internet routing delivers each packet to the nearest announcement by its own measure. A failed site withdraws its route, and traffic moves to the others. More in “DNS, GeoDNS and anycast for system design” name servers that answer; a policy store with weights, location maps and latency maps; health checkers in several independent networks; and a pipeline that turns policy and health into versioned snapshots for every name server. The hard part is who may call a region down, and what the name servers answer when everything behind them fails.

Requirements and estimate: TTLs set the load and the failover time

A company serves its API and app from four regions, Mumbai, Singapore, Frankfurt and Virginia, to users on every continent. Today a regional outage needs an engineer to change DNS by hand, and the change reaches users only as their caches expire. Ask first:

Functional requirements: answer A and AAAA queries by each name’s policy; health checks for every endpoint; weights and drains; a default answer for networks the maps do not know; allowed regions per name; versioned changes with rollback. Non-functional requirements: an answer within a few milliseconds from a nearby site; no outage when any one site, region, prober network or the control planecontrol plane: The part of a system that decides and distributes configuration, such as routes, certificates and policies, to the parts that carry the traffic, which together are called the data plane. More in “Service discovery, sidecars and service meshes” fails; and never an empty answer for a name that has a healthy region.

The first estimate, to change and run again:

# The first estimate for a DNS-based global traffic manager. Every input is an assumption, written down so that
# you can change it and run the script again.

RESOLVERS = 400_000        # recursive resolver addresses that ask for our names
NAMES_WARM = 10            # of our 40 names, how many an average resolver keeps in its cache
RECORD_TYPES = 2           # A and AAAA: dual-stack clients ask for both
TTL = 60                   # seconds, for the records that fail over
RESPONSE_BYTES = 200       # one answer: two addresses plus the EDNS record
SITES, SERVERS_PER_SITE = 16, 4
FLOOD_FACTOR = 100         # spare capacity for floods, as a multiple of the legitimate rate

ENDPOINTS = 12             # 4 regions x 3 load-balancer addresses
CHECKERS = 24              # 8 networks x 3 probers
INTERVAL = 10              # seconds between probes from each prober
FAILURES = 3               # failed probes in a row before an endpoint counts as down
PROBE_TIMEOUT = 4          # seconds a prober waits before calling a probe failed
AGREE, PUSH = 5, 5         # seconds to combine the votes, and to reach every name server (p99)

queries = RESOLVERS * NAMES_WARM * RECORD_TYPES / TTL
servers = SITES * SERVERS_PER_SITE
print("DNS queries")
print(f"  {RESOLVERS:,} resolvers x {NAMES_WARM} names x {RECORD_TYPES} record types / {TTL} s TTL "
      f"= {queries:,.0f} queries a second")
print(f"  answers out: {queries * RESPONSE_BYTES / 1e6:.1f} MB a second "
      f"({queries * RESPONSE_BYTES * 8 / 1e6:.0f} Mbit/s)")
print(f"  per name server ({SITES} sites x {SERVERS_PER_SITE}): {queries / servers:,.0f} a second; "
      f"with {FLOOD_FACTOR}x flood headroom: {queries / servers * FLOOD_FACTOR:,.0f}")

print("Health checks")
print(f"  {CHECKERS} probers x {ENDPOINTS} endpoints / {INTERVAL} s = {CHECKERS * ENDPOINTS / INTERVAL:.1f} probes "
      f"a second; each endpoint sees {CHECKERS / INTERVAL:.1f} a second")

detect_best = (FAILURES - 1) * INTERVAL + PROBE_TIMEOUT
detect_worst = FAILURES * INTERVAL + PROBE_TIMEOUT
print("Failover time for a client that obeys the TTL")
print(f"  notice: {FAILURES} failed probes {INTERVAL} s apart, plus a {PROBE_TIMEOUT} s timeout: "
      f"{detect_best} to {detect_worst} s")
print(f"  agree on the vote and push the new answers: {AGREE} s + {PUSH} s")
print(f"  cached answers expire: up to {TTL} s")
print(f"  worst case: {detect_worst + AGREE + PUSH + TTL} s; "
      f"the TTL is {TTL / (detect_worst + AGREE + PUSH + TTL):.0%} of it")

DNS queries
  400,000 resolvers x 10 names x 2 record types / 60 s TTL = 133,333 queries a second
  answers out: 26.7 MB a second (213 Mbit/s)
  per name server (16 sites x 4): 2,083 a second; with 100x flood headroom: 208,333
Health checks
  24 probers x 12 endpoints / 10 s = 28.8 probes a second; each endpoint sees 2.4 a second
Failover time for a client that obeys the TTL
  notice: 3 failed probes 10 s apart, plus a 4 s timeout: 24 to 34 s
  agree on the vote and push the new answers: 5 s + 5 s
  cached answers expire: up to 60 s
  worst case: 104 s; the TTL is 58% of it

The query rate is set by resolvers and TTLs, not by users: a resolverrecursive resolver: The DNS server that answers a device's question in full. It follows referrals from the root down to the servers authoritative for the name, and caches every answer for its TTL so the next user skips the walk. More in “DNS, GeoDNS and anycast for system design” that keeps a name warm asks once per TTLTTL (DNS): Time to live: the number of seconds a resolver or a client may keep a DNS answer in its cache before it must ask again. A change reaches a user only once the cached copy's TTL has run out. More in “DNS, GeoDNS and anycast for system design”, however many people it serves. The failover budgetfailover budget: The time from a region's failure until the last client that obeys the TTL has moved away: noticing the failure, agreeing on it, pushing new answers to every name server, and waiting for the cached answers to expire. is mostly the TTL, which is why the TTL of a name that fails over is chosen together with its health checks.

The first design answers from a validated snapshot

Each part of the design has one job:

• Anycast name servers: the same addresses announced from every site, so a query reaches a nearby site and a failed site drops out of routing.
• Policy store: for each name, its regions, weights, latency and location maps, allowed regions and default.
• Health checkers: probers in several independent networks that test every endpoint on a schedule.
• Health aggregator: turns many probes into one verdict per endpoint by a vote across networks.
• Snapshot builder: joins policy and verdicts into one versioned snapshot of every answer, and validates it.
• Distribution: pushes each snapshot to every site, where it replaces the old one only after a local check.
• Last valid snapshot: what a name server keeps answering from when nothing newer and valid arrives.

Follow one Mumbai outage through these parts:

The deep dives that decide whether the design works:

1. How many networks must agree before a region counts as down, and when may it come back?
2. How does one query become an answer: allowed regions, health, latency, weights and defaults?
3. What must a snapshot satisfy before any name server uses it, and how is an old one kept from overwriting a new one?
4. What do the name servers answer when the health data or the control plane stops?
5. How long do users keep reaching a failed region, given TTLs and clients that ignore them?
6. How do names that must stay in one country fail over?

An answer is judged on requirements and estimates, steering, health checking, the snapshot pipeline, behaviour under failure and operations.

The API, and one snapshot that holds every answer Pro: opens with a Pro, Premium or Ultimate pass