Redundancy, failover and static stability

Redundancy means keeping more than one of everything that matters, so that losing one copy does not stop the service. Failover is what happens when a copy fails: its work moves to the copies that remain, either at once, as a load balancer does for identical servers, or by promoting a standby, as a database does for its primary. Static stability is the property that makes failover trustworthy: the system keeps working through a failure without having to change anything, because the spare capacity is already running and every server already holds what it needs. The opposite design, which plans to launch servers or fetch configuration after something breaks, depends on machinery that is likely to be impaired at that very moment. This lesson sizes spare capacity across zones, separates control planes from data planes, compares constant work with failure modes, and keeps health checks from causing the outages they should detect.

How much spare capacity: surviving the loss of a zone

Cloud regions are divided into zones, separate locations built so that a power, cooling or network failure in one is unlikely to take the others down with it (Compute Engine documentation), and every AWS region has at least three (AWS documentation). A service spread over three zones survives the loss of one only if the other two can carry the whole peak. That is the arithmetic people get wrong: each zone needs half of the peak, not a third, so the fleet holds 150 % of the peak and runs at about two thirds of its capacity at a normal peak. The Builders’ Library describes exactly this sizing, three zones overprovisioned by 50 %, each running at about 66 % of the load it was tested for (static stability using Availability Zones).

In general, a service in z zones that must survive the loss of k of them needs each zone to carry a share of 1 ÷ (z − k) of the peak. The script turns that into servers for a peak of 30,000 requests a second and servers that handle 500 each.

# How much capacity must each zone hold so that the service still carries its peak after losing zones?
# A zone that may fail must not be needed: the zones that remain carry the whole peak.
import math

PEAK = 30_000  # requests a second at the daily peak
PER_SERVER = 500  # requests a second one server handles at its load-tested limit

print(f"Peak {PEAK:,} requests a second; one server handles {PER_SERVER} at its load-tested limit")
print(f"{'zones':>5} {'survives':>9} {'each zone':>10} {'fleet':>7} {'busy, all up':>13} {'servers/zone':>13} {'servers':>8}")
for zones in (2, 3, 4, 5):
    for lost in (1, 2):
        if lost >= zones:
            continue
        share = 1 / (zones - lost)  # of the peak, per zone
        per_zone = math.ceil(PEAK * share / PER_SERVER)
        print(
            f"{zones:>5} {f'{lost} zone' + ('s' if lost > 1 else ''):>9} {share:>10.0%} {zones * share:>7.0%}"
            f" {1 / (zones * share):>13.0%} {per_zone:>13} {zones * per_zone:>8}"
        )

# A deploy takes 10 % of each zone's servers out at a time. To lose a zone during a deploy, size for both.
zones, lost, out_for_deploy = 3, 1, 0.10
per_zone = math.ceil(PEAK / (zones - lost) / PER_SERVER / (1 - out_for_deploy))
print(f"\n3 zones, survive 1 zone while 10 % of servers are out for a deploy: {per_zone} servers per zone, "
      f"{zones * per_zone} in all, busy at {PEAK / (zones * per_zone * PER_SERVER):.0%} of their limit at the peak")

Peak 30,000 requests a second; one server handles 500 at its load-tested limit
zones  survives  each zone   fleet  busy, all up  servers/zone  servers
    2    1 zone       100%    200%           50%            60      120
    3    1 zone        50%    150%           67%            30       90
    3   2 zones       100%    300%           33%            60      180
    4    1 zone        33%    133%           75%            20       80
    4   2 zones        50%    200%           50%            30      120
    5    1 zone        25%    125%           80%            15       75
    5   2 zones        33%    167%           60%            20      100

3 zones, survive 1 zone while 10 % of servers are out for a deploy: 34 servers per zone, 102 in all, busy at 59% of their limit at the peak

More zones make redundancy cheaper: surviving one lost zone costs 100 % extra with two zones, 50 % with three, 33 % with four and 25 % with five. Surviving two lost zones is far more expensive, 200 % extra with three zones. Real fleets also lose capacity for planned reasons. Google’s SRE book states capacity needs as intents such as “N + 2 redundancy” in each region (SRE book, chapter 18); two spare units cover, for example, one failure while another unit is down for maintenance. The last line applies the same idea to zones: surviving a lost zone while 10 % of the servers are out for a deploy takes 34 servers per zone instead of 30.

Redundancy helps only against failures that do not hit every copy at once. Zones protect against power and network failures in one place; they do not protect against a bad configuration or a bad release pushed to every zone in the same minute. That is why careful teams roll changes out one zone at a time: a defect then fails one copy, not all.

Failover: active-active and active-standby

Active-active suits servers that keep no state of their own. Every zone serves traffic all the time; when a zone fails, its servers fail their health checks, and the load balancer sends their traffic to the zones that remain. The Builders’ Library’s example, a load-balanced fleet in three zones sized as above, needs no action at all when a zone is impaired. Active-standby suits things with one writer, such as a database primary: all writes go to the primary, and a warm standby in another zone is ready to take over. Its managed database example fails over by promoting the standby and pointing the database’s DNS name at it, and clusters with a leader elect the new leader from standbys that are already running, rather than starting a new node during the failure.

Both kinds of failover share one rule: exercise it. The Builders’ Library warns that a path used only during failures is rarely tested and tends to fail when it is finally needed, and suggests turning such fallback into failover that runs all the time in production, for example by serving from both copies routinely (Avoiding fallback). A failover you have never run is a hope, not a plan.

Static stability: keep working without the control plane

Most systems have two parts. The control plane makes changes: it creates, updates and deletes resources, such as launching a server, creating a storage bucket or changing a route. The data plane does the system’s actual work: a running server answering requests, reads and writes to a disk, a DNS server answering queries. AWS designs its data planes with deliberately fewer moving parts than its control planes, which are complex workflows with databases behind them, so failures are statistically less likely in the data plane (control planes and data planes).

A design is statically stable when it keeps working while a dependency is impaired: it may not see new changes, but everything it was doing before keeps working (static stability using Availability Zones). A running EC2 virtual machine is the textbook case: once launched, it needs no control plane to keep running, and its host already holds the routing information its packets need. The same idea applies to how you recover. A plan that says “if a zone fails, the autoscaler launches more servers in the others” makes recovery depend on the control plane at the moment it is most stressed, and the AWS whitepaper puts the principle plainly: the most reliable recovery mechanisms are the ones that need the fewest changes (static stability).

The simulation compares the two designs after a zone fails at the peak: one holds 34 % of the peak per zone and launches servers after the failure, which take 8 minutes once the request succeeds; the other holds 50 % per zone. The third row assumes the same event impairs the control plane for 25 minutes, an assumption you can change.

# One of three zones fails at minute 0, at the daily peak. Two designs:
#   scale up after the failure: each zone holds 34 % of the peak, and the autoscaler asks the control plane for new
#     servers, which take 8 minutes to launch and warm up once the request succeeds;
#   statically stable: each zone already holds 50 % of the peak, so the two zones left carry it without any change.
# The same event may also impair the control plane (the launch API) for a while: the third scenario assumes
# 25 minutes. Minute by minute, what share of the peak traffic is served?
PEAK = 1.0
BOOT_MINUTES = 8


def served(minute, per_zone, scales_up, control_plane_back):
    capacity = 2 * per_zone  # one zone of three is gone
    if scales_up:
        launched_at = max(0, control_plane_back)  # the launch request succeeds once the control plane answers
        if minute >= launched_at + BOOT_MINUTES:
            capacity = PEAK  # the new servers fill the gap
    return min(PEAK, capacity)


scenarios = [
    ("scale up, control plane works", 0.34, True, 0),
    ("scale up, control plane down 25 min", 0.34, True, 25),
    ("statically stable (50 % per zone)", 0.50, False, 25),
]
minutes = [0, 5, 8, 10, 20, 30, 40]
print(f"{'design':<37}" + "".join(f"{f'{m} min':>8}" for m in minutes) + f"{'lost in 60 min':>16}")
for name, per_zone, scales_up, back in scenarios:
    shares = [served(m, per_zone, scales_up, back) for m in range(60)]
    lost = sum(PEAK - s for s in shares) / 60  # share of an hour's peak traffic that was refused
    row = "".join(f"{served(m, per_zone, scales_up, back):>8.0%}" for m in minutes)
    print(f"{name:<37}{row}{lost:>16.1%}")

design                                  0 min   5 min   8 min  10 min  20 min  30 min  40 min  lost in 60 min
scale up, control plane works             68%     68%    100%    100%    100%    100%    100%            4.3%
scale up, control plane down 25 min       68%     68%     68%     68%     68%     68%    100%           17.6%
statically stable (50 % per zone)        100%    100%    100%    100%    100%    100%    100%            0.0%

When the control plane works, the reactive design refuses 32 % of the traffic for 8 minutes, 4.3 % of the hour’s peak traffic. When the control plane is impaired too, the gap lasts 33 minutes and costs 17.6 %. The statically stable design loses nothing in either case, because it never asks for anything. The same thinking extends beyond capacity:

• Keep the last good configuration. A server that reads its routing rules or feature flags from a configuration service should cache the last version it loaded and keep serving with it when that service is down.
• Push what servers need before they need it. The Builders’ Library’s example is credentials delivered to every server ahead of time and valid for hours, so that requests keep working even if delivery stops for a while.
• Keep standbys warm and running, and avoid recovery steps that need the thing that has failed.

Constant work: the same work in good times and bad

A coffee urn makes coffee for one person or for a hundred with the same effort: it does a constant amount of work whoever shows up. The Builders’ Library uses the urn to describe constant-work systems, which do the same operations in all conditions, have no special modes, and if anything do less work under stress (constant work). Its examples are concrete. Every few seconds, Amazon Route 53’s health check aggregators send the Route 53 DNS servers a fixed-size table of health statuses, so a zone failure that flips thousands of statuses changes the contents of the table, not the amount of work. AWS Hyperplane nodes fetch their whole configuration file every few seconds, even when nothing has changed.

The alternative sends only changes, which is cheaper on a quiet day and much busier on a bad one. The example compares the two for 10,000 health-checked targets when a zone failure takes 3,300 of them down at once, with 1 % of messages lost on the way.

# Health checkers watch 10,000 targets and tell the DNS servers which are healthy. Two ways to send the news:
#   changes only: one update per target whose state changed; the DNS server applies each update in 5 ms;
#   full table: every second, a fixed-size table of all 10,000 states, applied in one 20 ms step, changed or not.
# Normally 5 targets change state a second. At 30 s a zone fails and 3,300 targets go down at once. Also, 1 % of the
# messages are lost on the way, and a lost change stays lost until that target changes state again.
import random

TARGETS, SECONDS, FAIL_AT = 10_000, 60, 30
UPDATE_MS, TABLE_MS = 5, 20


def run(full_table, seed=2):
    rng = random.Random(seed)
    truth = [True] * TARGETS  # what the health checkers see
    seen = [True] * TARGETS  # what the DNS server believes
    backlog = []  # updates waiting to be applied (changes-only design)
    busiest = 0
    for second in range(SECONDS):
        changed = rng.sample(range(TARGETS), 5)
        if second == FAIL_AT:
            changed += range(3_300)  # a whole zone goes down
        for t in changed:
            truth[t] = not truth[t] if second != FAIL_AT or t >= 3_300 else False
        if full_table:
            work_ms = TABLE_MS  # the same work every second
            if rng.random() > 0.01:  # this second's table arrived
                seen = truth[:]
        else:
            backlog += [(t, truth[t]) for t in changed if rng.random() > 0.01]
            applied, backlog = backlog[: 1000 // UPDATE_MS], backlog[1000 // UPDATE_MS :]
            for t, state in applied:
                seen[t] = state
            work_ms = len(applied) * UPDATE_MS
        busiest = max(busiest, work_ms)
        wrong = sum(1 for a, b in zip(truth, seen) if a != b)
        if second in (FAIL_AT - 1, FAIL_AT, FAIL_AT + 5, FAIL_AT + 15, SECONDS - 1):
            print(f"  {second:>3} s: DNS server busy {work_ms:>4} ms of the second; {wrong:>5,} targets shown wrongly")
    return busiest


for name, full in [("changes only", False), ("full table every second", True)]:
    print(f"{name}:")
    busiest = run(full)
    print(f"  busiest second: {busiest} ms of work\n")

changes only:
   29 s: DNS server busy   25 ms of the second;     1 targets shown wrongly
   30 s: DNS server busy 1000 ms of the second; 3,052 targets shown wrongly
   35 s: DNS server busy 1000 ms of the second; 2,073 targets shown wrongly
   45 s: DNS server busy 1000 ms of the second;   175 targets shown wrongly
   59 s: DNS server busy   25 ms of the second;    41 targets shown wrongly
  busiest second: 1000 ms of work

full table every second:
   29 s: DNS server busy   20 ms of the second;     0 targets shown wrongly
   30 s: DNS server busy   20 ms of the second;     0 targets shown wrongly
   35 s: DNS server busy   20 ms of the second;     0 targets shown wrongly
   45 s: DNS server busy   20 ms of the second;     0 targets shown wrongly
   59 s: DNS server busy   20 ms of the second;     0 targets shown wrongly
  busiest second: 20 ms of work

Sending changes, the DNS server spends whole seconds on nothing but updates: the 3,300 changes take about 16 seconds to apply, and in the meantime it shows up to 3,000 dead targets as healthy. Lost messages leave errors behind for good: 41 targets are still wrong at the end of the minute. The full table costs 20 ms every second, the same in the failure as before it, shows every change within a second, and repairs a lost table with the next one: it is always in “repair everything” mode. The price is waste, sending 10,000 states when 5 changed, which is negligible for configuration and status but not for everything. The same article notes that for web servers, scaling elastically is often the better fit, because the savings are large and the scaling runs often enough to stay well tested.

Health checks that do not take the fleet down

Health checks decide which servers get traffic, so a bad one can cause the outage it should prevent. The Builders’ Library describes four kinds: liveness checks that a process answers at all; local checks of resources the server does not share with its peers, such as its disk; dependency checks of the services it calls; and anomaly detection, which compares each server with its peers (implementing health checks). The danger is a check whose failure is correlated across servers. If every server’s check calls the same database and the database slows down, every server fails its check at the same moment, and a load balancer that obeys removes the whole fleet, including servers that could have served cached pages.

Three habits avoid it:

1. Keep fast load-balancer checks local (liveness and local checks), and leave dependency checks to monitoring that alarms, or acts slowly with a cap on how many servers it may remove.
2. Fail open. When every server looks unhealthy, the check is more likely wrong than the fleet; the Builders’ Library notes that some load balancers then send traffic to all servers.
3. Answer health checks first under load. A server that is too busy to answer its check is removed, and its load lands on the others, which can start a downward spiral.

Kubernetes separates the two questions clearly. A failing liveness probe restarts the container, and a failing readiness probe only stops sending it traffic; its documentation warns that badly made liveness probes can cause cascading failures, and suggests a liveness probe that checks the app itself while the readiness probe also checks the back-end services it needs (Kubernetes probes).

Practice

import math


def zone_share(zones, zones_lost):
    """The share of the peak each zone must carry so that the zones left after a failure carry it all."""
    # Replace this line with your code.
    return 0


def servers_per_zone(peak_rps, per_server_rps, zones, zones_lost, out_for_deploy=0.0):
    """Servers each zone needs to survive the loss of `zones_lost` zones, with a share out for a deploy."""
    # Replace this line with your code.
    return 0


def busy_at_peak(peak_rps, per_server_rps, servers):
    """How busy all the servers are at the peak when none has failed (a fraction)."""
    # Replace this line with your code.
    return 0

import math

from zones import busy_at_peak, servers_per_zone, zone_share


def raises_value_error(function, *args):
    try:
        function(*args)
    except ValueError:
        return True
    return False


def test_zone_share():
    """the zones that remain share the whole peak"""
    assert zone_share(3, 1) == 0.5
    assert zone_share(2, 1) == 1
    assert math.isclose(zone_share(4, 1), 1 / 3)
    assert math.isclose(zone_share(5, 2), 1 / 3)


def test_no_zone_left():
    """losing every zone cannot be planned for"""
    assert raises_value_error(zone_share, 2, 2)
    assert raises_value_error(zone_share, 3, 4)
    assert not raises_value_error(zone_share, 3, 2)


def test_servers_per_zone():
    """servers for the zone's share of the peak, rounded up"""
    assert servers_per_zone(30_000, 500, 3, 1) == 30
    assert servers_per_zone(30_000, 500, 4, 1) == 20
    assert servers_per_zone(30_000, 500, 3, 2) == 60
    assert servers_per_zone(10_001, 1_000, 2, 1) == 11


def test_servers_with_a_deploy():
    """servers out for a deploy must not be needed either"""
    assert servers_per_zone(30_000, 500, 3, 1, out_for_deploy=0.1) == 34
    assert servers_per_zone(12_000, 400, 3, 1, out_for_deploy=0.25) == 20


def test_busy_at_peak():
    """with every zone up, the spare capacity shows as low utilisation"""
    assert math.isclose(busy_at_peak(30_000, 500, 90), 2 / 3)
    assert math.isclose(busy_at_peak(30_000, 500, 120), 0.5)
    assert math.isclose(busy_at_peak(10_000, 1_000, 40), 0.25)

Interview questions

Warm-up (fresher to mid level): what does static stability mean? A statically stable system keeps working when a dependency fails, without having to change anything. It may stop seeing new changes, such as new servers or new configuration, but whatever it was doing before keeps working. In practice that means the spare capacity for a zone failure is already running instead of being launched afterwards, servers keep the last configuration and credentials they loaded, and recovery never depends on a control plane, the part that makes changes, which is more complex and more likely to be impaired during the same event. The test is simple: list the steps your recovery needs, and remove every step that asks some system to do something new.

Key takeaways

• To survive the loss of k of z zones, each zone must carry 1 ÷ (z − k) of the peak: half of it with three zones and one loss, so the fleet holds 150 % of the peak.
• Add room for planned work too: N + 2, or surviving a zone while a deploy has servers out.
• Redundancy helps only against failures that do not hit every copy at once: roll changes out zone by zone.
• Active-active fleets fail over without any action; active-standby stores promote a standby that is already running. Exercise both regularly.
• Statically stable designs keep working without the control plane: capacity already provisioned, last good configuration cached, credentials pushed ahead of time.
• Constant-work designs do the same work in all conditions, and repair themselves on the next cycle.
• Keep load-balancer health checks local, fail open when every server looks unhealthy, and answer health checks before other work.