Timeouts, retries, backoff and jitter

A call to another service can succeed, fail, or simply not come back. A timeout decides how long you wait before giving up. A retry sends the call again in the hope that the failure was brief. Exponential backoff waits longer before each new attempt, and jitter makes every client wait a slightly different, random time, so that clients that failed together do not all return together. Used well, the four turn short blips into nothing a user notices. Used carelessly, they cause outages of their own: a missing timeout ties up every thread behind one slow dependency, retries at several layers multiply the load on a service that is already struggling, and retries without jitter come back in waves that knock it over again. This lesson sets each one from numbers you can measure, then adds the two controls that keep retries safe: a deadline that travels with the request, and a retry budget.

Every remote call needs a timeout

Without a timeout, a call to a dependency that has stopped answering waits for as long as the connection stays open, which can be minutes. Defaults rarely protect you. The Python Requests library does not time out unless you pass a timeout, and its documentation warns that such code may hang for minutes or more (Requests documentation). gRPC sets no deadline by default, so a client can wait, in the words of its guide, “effectively forever” (gRPC deadlines). Every waiting call holds a thread, a connection or some memory, and by Little’s law the number held is the call rate times the waiting time. A dependency that slows down turns into a caller that runs out of threads.

So the question is not whether to set a timeout but which value. A timeout that is too long holds resources for longer during a failure. One that is too short gives up on calls that would have succeeded, and those calls are usually retried, which adds load. The Amazon Builders’ Library describes a way to choose from data: decide what share of good calls you are willing to cut off, say 0.1 %, and set the timeout at that percentile of the dependency’s measured latency, the p99.9 (Builders’ Library). The script below does that for a seeded model of a dependency whose calls mostly take about 20 ms, with 2 % of them stuck behind a pause of up to 800 ms. Then it asks what each candidate timeout would cost if the dependency hung completely.

# Choose a timeout from the dependency's measured latency, then see what it costs.
# A seeded model of one dependency: most calls take about 20 ms, and 2 % hit a pause of 150-800 ms
# (a garbage-collection pause, a cold cache, a compaction), the kind of tail real services have.
import bisect
import math
import random

rng = random.Random(7)
CALLS = 200_000
latencies = []
for _ in range(CALLS):
    ms = rng.lognormvariate(math.log(20), 0.5)
    if rng.random() < 0.02:
        ms += rng.uniform(150, 800)
    latencies.append(ms)
latencies.sort()


def percentile(p):
    return latencies[min(CALLS - 1, int(CALLS * p / 100))]


print(f"Latency of {CALLS:,} calls")
for p in (50, 90, 99, 99.9, 99.99):
    print(f"  p{p:<6}{percentile(p):6.0f} ms")
print(f"  max    {latencies[-1]:6.0f} ms")

RATE = 400  # calls a second that one instance makes to this dependency
POOL = 200  # threads (or connections) the instance has for these calls
print(f"\nTimeouts for one instance making {RATE} calls a second with {POOL} threads:")
print(f"{'timeout':>9}  {'calls cut off':>13}  threads held if the dependency hangs")
for timeout in (50, 100, 250, 500, 800, 1000, 2000):
    cut = (CALLS - bisect.bisect_right(latencies, timeout)) / CALLS
    held = RATE * timeout / 1000  # Little's law: during a hang every call waits the whole timeout
    note = "within the pool" if held <= POOL else f"{held / POOL:.1f} times the pool"
    print(f"{timeout:>6} ms  {cut:>13.2%}  {held:>5.0f} ({note})")

Latency of 200,000 calls
  p50        20 ms
  p90        40 ms
  p99       498 ms
  p99.9     788 ms
  p99.99    826 ms
  max       857 ms

Timeouts for one instance making 400 calls a second with 200 threads:
  timeout  calls cut off  threads held if the dependency hangs
    50 ms          5.25%     20 (within the pool)
   100 ms          2.02%     40 (within the pool)
   250 ms          1.74%    100 (within the pool)
   500 ms          0.99%    200 (within the pool)
   800 ms          0.07%    320 (1.6 times the pool)
  1000 ms          0.00%    400 (2.0 times the pool)
  2000 ms          0.00%    800 (4.0 times the pool)

Read the two columns against each other. A 250 ms timeout looks generous next to a 20 ms median, yet it cuts off 1.74 % of calls that would have succeeded, because the slow 2 % sit far out in the tail. The p99.9 is 788 ms, and a timeout of 800 ms cuts off only 0.07 %. But if the dependency hangs, every call waits the whole 800 ms, and 400 calls a second then hold 320 threads, more than the 200 this instance has. A timeout alone cannot protect the caller here: it also needs a cap on how many calls to this one dependency may be waiting at once, the bulkhead of the next lesson.

Two refinements matter in practice. Give the connection its own, shorter timeout, because connecting to a healthy server takes a round trip or two while the time to an answer depends on the work. And when the p99.9 is close to the median, add some padding: a timeout that tight fires on ordinary noise. For clients that reach you over the internet, allow for a slow mobile network as well.

A deadline travels with the request

A timeout belongs to one call; a request crosses many calls. If the edge gives a user’s request one second and every service below starts its own fresh timeout, a service deep in the chain can keep working long after the user has given up. Google’s SRE book finds this pattern in many cascading outages, servers busy with requests whose clients have already left, and its remedy is deadline propagation: set the deadline once, near the top, and pass what is left of it with every call below (SRE book, chapter 22). gRPC’s Java and Go libraries do this for you, and they send the time remaining rather than a clock time, so two servers whose clocks disagree still agree on how long is left (gRPC deadlines). A server that receives a request whose deadline has passed drops it, and a long request checks the time left before each expensive step.

// A deadline set once at the edge and passed down a chain of services, on a simulated clock.
// The edge gives each request 1,000 ms. Every service passes on what is left after its own work and 10 ms of network
// time, and a service that receives a request whose deadline has passed drops it instead of working for nobody.
const DEADLINE_MS = 1000;
const NETWORK_MS = 10;
const chain = [
  ["gateway", 20],
  ["orders", 150],
  ["payments", 120],
];

for (const [label, queuedMs] of [
  ["a normal request", 0],
  ["the same request after 900 ms in a queue", 900],
]) {
  console.log(`${label === "a normal request" ? "" : "\n"}Deadline 1,000 ms, ${label}:`);
  let now = queuedMs;
  for (const [service, workMs] of chain) {
    const left = DEADLINE_MS - now;
    if (left <= 0) {
      console.log(`  ${service.padEnd(8)} gets it with ${left} ms left: dropped, the caller has given up`);
      break;
    }
    console.log(`  ${service.padEnd(8)} gets it with ${String(left).padStart(4)} ms left and works ${workMs} ms`);
    now += workMs + NETWORK_MS;
  }
}

// Retries inside the same deadline: each attempt gets at most 300 ms, and the wait between attempts is capped
// exponential backoff with full jitter (base 50 ms). A seeded generator stands in for Math.random.
function seeded(seed) {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const random = seeded(3);
const PER_TRY_MS = 300;
const MIN_USEFUL_MS = 100; // an attempt with less time than this cannot finish, so it is not sent
console.log(`\nA dependency that hangs: attempts of at most ${PER_TRY_MS} ms inside one 1,000 ms deadline`);
let now = 0;
for (let retry = 0; ; retry++) {
  const left = DEADLINE_MS - now;
  if (left < MIN_USEFUL_MS) {
    console.log(`  ${String(now).padStart(4)} ms: ${left} ms left, too little for another attempt: fail now`);
    break;
  }
  const budget = Math.min(PER_TRY_MS, left);
  console.log(`  ${String(now).padStart(4)} ms: attempt ${retry + 1} with a ${budget} ms timeout, which runs out`);
  now += budget;
  const wait = Math.round(random() * Math.min(1000, 50 * 2 ** retry));
  if (DEADLINE_MS - now - wait < MIN_USEFUL_MS) {
    console.log(`  ${String(now).padStart(4)} ms: a ${wait} ms backoff would leave ${DEADLINE_MS - now - wait} ms: fail now`);
    break;
  }
  console.log(`  ${String(now).padStart(4)} ms: back off ${wait} ms`);
  now += wait;
}

Deadline 1,000 ms, a normal request:
  gateway  gets it with 1000 ms left and works 20 ms
  orders   gets it with  970 ms left and works 150 ms
  payments gets it with  810 ms left and works 120 ms

Deadline 1,000 ms, the same request after 900 ms in a queue:
  gateway  gets it with  100 ms left and works 20 ms
  orders   gets it with   70 ms left and works 150 ms
  payments gets it with -90 ms left: dropped, the caller has given up

A dependency that hangs: attempts of at most 300 ms inside one 1,000 ms deadline
     0 ms: attempt 1 with a 300 ms timeout, which runs out
   300 ms: back off 36 ms
   336 ms: attempt 2 with a 300 ms timeout, which runs out
   636 ms: back off 4 ms
   640 ms: attempt 3 with a 300 ms timeout, which runs out
   940 ms: a 91 ms backoff would leave -31 ms: fail now

The first chain is the normal case: each service sees less time than the one before. In the second, the request waited 900 ms in a queue before the gateway took it. The payments service drops it, but the orders service still spent 150 ms on a request with 70 ms left, because it did not check the deadline before its expensive step. The third part shows retries inside a deadline: three attempts of 300 ms fit into one second, and the client stops when the next backoff would leave no time for an answer, instead of sending an attempt that cannot finish.

Retry only what can succeed, and only what is safe

A retry helps only when two things are true: the failure may not happen again, and sending the request twice does no harm. The first rules out failures that will repeat. A request rejected as invalid fails the same way every time, and the Azure retry pattern says to cancel the operation in that case, to retry at once only for a rare fault such as a corrupted packet, and to retry after a delay for the common busy and connectivity failures (Azure retry pattern). The second is about side effects. HTTP calls a method idempotent when sending it several times has the same intended effect as sending it once; GET, HEAD, OPTIONS, PUT and DELETE are, POST is not. RFC 9110 asks clients not to resend a non-idempotent request by themselves unless they can tell that repeating it is harmless, or that the first one never took effect (RFC 9110, section 9.2.2). A timeout is the dangerous case, because it does not say whether the server did the work: a payment that timed out may have gone through. An idempotency key, a unique id the server remembers for each operation, makes such a call safe to repeat.

Six common failures, and whether to retry them:

• A GET that timed out, or whose connection reset: retry, with backoff. Reading twice changes nothing.
• 503 Service Unavailable with Retry-After: 2: retry after at least 2 seconds. The server says how long it expects to be unavailable.
• 429 Too Many Requests: retry after the wait it asks for. The server is limiting this client’s rate.
• A POST that creates a payment and timed out, without an idempotency key: do not retry. The payment may already have been made.
• 400 or 422, a request the server calls invalid: do not retry. The same request fails the same way.
• 401 or 403: not as it is. Refresh the credentials once, and if that fails, report the error.

Retry-After is the server’s way to say when to come back: a number of seconds or a date, sent with a 503 to say how long the service expects to be unavailable (RFC 9110, section 10.2.3), and allowed with a 429 Too Many Requests too (RFC 6585). Respect it as a lower bound for the next attempt. Log early failed attempts quietly and only the last one as an error, as the Azure pattern suggests, so that a retry that worked does not wake anyone up.

Exponential backoff, capped, with jitter

Waiting before a retry gives a busy dependency time to recover, and waiting longer each time gives it more. Exponential backoff doubles the wait after every failed attempt: with a base of 10 ms, the waits are 10, 20, 40, 80 ms and so on, up to a cap that stops them from growing for ever. Backoff alone has a flaw that the next run shows: clients that failed at the same moment wait exactly the same times and come back at the same moment again. Jitter fixes it by making each wait random. Marc Brooker’s post on the AWS Architecture Blog compares three variants (AWS Architecture Blog):

• Full jitter waits a random time between 0 and the capped exponential value.
• Equal jitter waits half of the capped value plus a random time up to the other half, so it never waits very little.
• Decorrelated jitter waits a random time between the base and three times the previous wait, capped.

The simulation below sends 1,000 clients at one server at the same instant, the way a job that fires on every host at the top of the minute does, or the end of a network blip. The server has 5 slots a millisecond, and rejecting a request still costs a twentieth of a slot, because it has to read the request and answer it.

# 1,000 clients call one server at the same moment: a job that fires on every host at the top of the minute, or the
# end of a network blip. The server has 5 slots a millisecond. Serving a request takes a slot; rejecting one with a
# 503 still costs a twentieth of a slot (it has to read the request and answer), so a big enough crowd leaves no
# slot for real work. Rejected clients retry after a delay, at most 8 attempts in all.
# Six retry strategies with the same seed: how much work, how many clients served, and how fast?
import random

CLIENTS = 1_000
SLOTS = 5.0  # work the server can do per millisecond
REJECT_COST = 0.05  # share of a slot that a rejection costs
ATTEMPTS = 8  # the first call and up to 7 retries
BASE, CAP = 10, 2_000  # backoff: first delay 10 ms, never longer than 2 s


def immediate(rng, retry, prev):
    return 1


def fixed(rng, retry, prev):
    return 100


def exponential(rng, retry, prev):
    return min(CAP, BASE * 2**retry)


def equal_jitter(rng, retry, prev):
    d = min(CAP, BASE * 2**retry)
    return d / 2 + rng.uniform(0, d / 2)


def full_jitter(rng, retry, prev):
    return rng.uniform(0, min(CAP, BASE * 2**retry))


def decorrelated(rng, retry, prev):
    return min(CAP, rng.uniform(BASE, prev * 3))


def served_in(arrivals):
    """Requests served in one millisecond: what is left of the slots after paying for the rejections."""
    if arrivals <= SLOTS:
        return arrivals
    return max(0, int((SLOTS - REJECT_COST * arrivals) / (1 - REJECT_COST)))


def run(strategy, seed=1):
    rng = random.Random(seed)
    due = {0: [(c, 0, BASE) for c in range(CLIENTS)]}  # millisecond -> [(client, retries so far, last delay)]
    calls = busiest = served = gave_up = last = 0
    t = 0
    while due:
        batch = due.pop(t, [])
        if batch:
            calls += len(batch)
            if t > 0:
                busiest = max(busiest, len(batch))
            rng.shuffle(batch)  # which requests get a slot is luck
            ok = served_in(len(batch))
            served += ok
            if ok:
                last = t
            for client, retries, prev in batch[ok:]:
                if retries + 1 == ATTEMPTS:
                    gave_up += 1
                    continue
                delay = max(1, round(strategy(rng, retries, prev)))
                due.setdefault(t + delay, []).append((client, retries + 1, delay))
        t += 1
    return calls, served, gave_up, busiest, last


print(f"{CLIENTS:,} clients at once, {SLOTS:.0f} slots a millisecond, at most {ATTEMPTS} attempts each")
print(f"{'strategy':<26}{'calls':>7}{'served':>8}{'gave up':>9}{'busiest retry ms':>18}{'last served':>13}")
for name, strategy in [
    ("retry at once", immediate),
    ("fixed 100 ms", fixed),
    ("exponential, no jitter", exponential),
    ("exponential, equal jitter", equal_jitter),
    ("exponential, full jitter", full_jitter),
    ("decorrelated jitter", decorrelated),
]:
    calls, served, gave_up, busiest, last = run(strategy)
    when = f"{last:,} ms" if served else "never"
    print(f"{name:<26}{calls:>7,}{served:>8,}{gave_up:>9,}{busiest:>12,} calls{when:>13}")

1,000 clients at once, 5 slots a millisecond, at most 8 attempts each
strategy                    calls  served  gave up  busiest retry ms  last served
retry at once               8,000       0    1,000       1,000 calls        never
fixed 100 ms                8,000       0    1,000       1,000 calls        never
exponential, no jitter      8,000       0    1,000       1,000 calls        never
exponential, equal jitter   5,792   1,000        0         223 calls       897 ms
exponential, full jitter    5,844     997        3         152 calls       784 ms
decorrelated jitter         4,580     998        2          70 calls       556 ms

The three strategies without randomness never serve a single client. All 1,000 come back in the same millisecond, the rejections use up every slot, and nobody gets through, eight attempts in a row. Exponential backoff does not change that, because 1,000 clients that wait exactly 10, 20 and 40 ms still arrive together. Any jitter changes everything: the busiest millisecond of retries falls from 1,000 calls to between 70 and 223, and nearly every client is served within a second. Between the jittered strategies the gaps are small. Here decorrelated jitter did about a fifth less work than full jitter, because its first retries spread over 10 to 30 ms instead of 0 to 10 ms; in Brooker’s simulations, which modelled contention differently, full jitter did the least work and decorrelated jitter finished slightly sooner. Two lessons hold in both models. Never retry without jitter, and give the jitter a window wide enough for the crowd: 1,000 clients at 5 a millisecond need at least 200 ms.

Libraries already do this, so check their settings rather than writing your own. The AWS SDKs’ standard retry mode makes at most 3 attempts by default, waits a full-jitter delay, and never waits more than 20 seconds (AWS SDK retry behavior). A gRPC retry policy sets the attempts, the first and the longest backoff and the multiplier, adds 20 % of jitter either way, and stops retrying once the server’s response headers have arrived (gRPC retry). Jitter is not only for retries either: timers, scheduled jobs and cache entries that expire together line up the same way, and the Builders’ Library adds jitter to all of them.

Retries multiply across layers

Each layer that retries multiplies the attempts of the layers above it. If a request passes three services that each try a failing call 3 times, the database at the bottom receives up to 3 × 3 × 3 = 27 calls for one user request.

The numbers grow quickly with depth. The SRE book’s example has three layers that each retry 3 times, 4 attempts each, and arrives at 64 database attempts for one user action (SRE book, chapter 22); the Builders’ Library works through five layers and arrives at 243 times the load, 3 to the fifth power. The formula is attempts per layer raised to the number of layers.

Worse, the extra load can outlive its cause. Bronson and his co-authors describe a web application that sends a database 280 queries a second and retries any query not answered within 1 second; the database answers within 100 ms below 300 queries a second (Metastable failures). A 10-second network outage queues a burst of requests and retries, the burst slows the database past 1 second, and from then on every query is retried: 560 a second, which keeps the database slow. The network is long fixed, but the system stays down until its load falls under 150 queries a second or its retries under 20 a second. The paper calls such a state, held in place by the extra work the system does to recover, a metastable failure.

The fix is structural. Retry at one layer only, ideally the one directly above the dependency that failed, and let the others fail fast. Google’s SRE book adds a signal for the way up: a layer that has used its retries answers with an error that means “overloaded, do not retry”, so that the layers above pass it on instead of retrying it (SRE book, chapter 21).

A retry budget caps the extra load

Even one layer of retries triples the load during an outage if every call is tried 3 times, which is the moment the dependency can least afford it. A retry budget puts a ceiling on that. The SRE book combines two limits: at most 3 attempts per request, and retries only while they are below 10 % of a client’s requests. Together they bring the worst case down from almost 3 times the load to about 1.1 times (SRE book, chapter 21). The book also suggests a budget per server process, such as 60 retries a minute, beyond which the process returns an error instead (chapter 22).

A token bucket is the usual way to keep such a budget. gRPC’s retry throttling takes a token for every failed call and gives back a fraction for every success, and pauses retries while fewer than half its tokens are left (gRPC retry). The AWS SDKs keep a retry quota of 500 tokens per client that retries draw down and successes refill. The example below keeps a budget of 10 % and drives a client through three phases: a healthy dependency, an outage and a slow recovery.

// A client that retries failed calls, with and without a retry budget, through three phases of a dependency's life.
// A seeded random generator decides which calls fail, so every run prints the same.

// mulberry32: a small seeded random generator (Math.random cannot be seeded).
function seeded(seed) {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// A retry budget as a token bucket: every request earns 0.1 token (at most 10 are kept) and every retry spends one,
// so retries stay near 10 % of requests however bad things get.
class RetryBudget {
  constructor(ratio = 0.1, max = 10) {
    this.ratio = ratio;
    this.max = max;
    this.tokens = max;
  }
  onRequest() {
    this.tokens = Math.min(this.max, this.tokens + this.ratio);
  }
  tryRetry() {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

// One request: up to `attempts` tries; with a budget, a retry is sent only if the budget has a token for it.
function request(failRate, random, attempts, budget) {
  if (budget) budget.onRequest();
  let sent = 0;
  for (let i = 0; i < attempts; i++) {
    if (i > 0 && budget && !budget.tryRetry()) break;
    sent++;
    if (random() >= failRate) return { sent, ok: true };
  }
  return { sent, ok: false };
}

const phases = [
  ["healthy, 1 % fail", 0.01],
  ["outage, all fail", 1.0],
  ["recovering, 20 % fail", 0.2],
];
const policies = [
  ["no retries", 1, false],
  ["3 attempts", 3, false],
  ["3 attempts, 10 % budget", 3, true],
];
const N = 10000; // requests in each phase
console.log("10,000 requests in each phase: calls the dependency receives per request, and requests that succeed");
console.log("policy".padEnd(24) + phases.map(([name]) => name.padStart(24)).join(""));
for (const [name, attempts, withBudget] of policies) {
  const random = seeded(7);
  const budget = withBudget ? new RetryBudget() : null;
  const cells = phases.map(([, failRate]) => {
    let sent = 0;
    let ok = 0;
    for (let i = 0; i < N; i++) {
      const r = request(failRate, random, attempts, budget);
      sent += r.sent;
      if (r.ok) ok++;
    }
    return `${(sent / N).toFixed(2)}x, ${((100 * ok) / N).toFixed(1)} % ok`.padStart(24);
  });
  console.log(name.padEnd(24) + cells.join(""));
}

// A check of the budget, as a plain assertion.
const budget = new RetryBudget();
let granted = 0;
for (let i = 0; i < 1000; i++) {
  budget.onRequest();
  if (budget.tryRetry()) granted++;
}
console.log(`\n1,000 failing requests in a row are granted ${granted} retries: the 10 saved tokens plus 0.1 a request`);
console.log(granted === 109 ? "check passed: retries stay near 10 % of requests" : "check FAILED");

10,000 requests in each phase: calls the dependency receives per request, and requests that succeed
policy                         healthy, 1 % fail        outage, all fail   recovering, 20 % fail
no retries                      1.00x, 98.8 % ok         1.00x, 0.0 % ok        1.00x, 80.2 % ok
3 attempts                     1.01x, 100.0 % ok         3.00x, 0.0 % ok        1.25x, 99.1 % ok
3 attempts, 10 % budget        1.01x, 100.0 % ok         1.10x, 0.0 % ok        1.10x, 88.3 % ok

1,000 failing requests in a row are granted 109 retries: the 10 saved tokens plus 0.1 a request
check passed: retries stay near 10 % of requests

When the dependency is healthy the budget changes nothing, because retries are rare. During the outage, three attempts per request triple the load, and the budget holds it at 1.10 times. The recovery phase shows the price: with 20 % of calls failing, the client needs more retries than 10 % allows, so 88.3 % of requests succeed with the budget against 99.1 % without it. That is the trade a budget makes on purpose. It protects the dependency first and the client’s success rate second, so choose the ratio for the failures you expect: short blips fit easily into 10 %, and a higher ratio buys back success during partial failures at the cost of more load during outages.

Practice

import random

RETRYABLE_STATUSES = {408, 429, 500, 502, 503, 504}
IDEMPOTENT_METHODS = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE"}


def backoff_delay(retry, base_ms, cap_ms, rand=random.random):
    """Milliseconds to wait before retry number `retry`: full jitter over min(cap_ms, base_ms * 2 ** retry)."""
    # Replace this line with your code.
    return 0


def retry_delays(attempts, base_ms, cap_ms, rand=random.random):
    """The delay before each retry of a call whose attempts all fail."""
    # Replace this line with your code.
    return []


def should_retry(status, method, has_idempotency_key=False):
    """Whether a failed call (status None: a timeout or a failed connection) may be retried."""
    # Replace this line with your code.
    return False

from backoff import backoff_delay, retry_delays, should_retry


def test_window_doubles():
    """the window doubles with every retry, and rand picks a point in it"""
    assert backoff_delay(0, 100, 10_000, lambda: 0.5) == 50
    assert backoff_delay(1, 100, 10_000, lambda: 0.5) == 100
    assert backoff_delay(3, 100, 10_000, lambda: 0.5) == 400
    assert backoff_delay(2, 50, 10_000, lambda: 0.25) == 50


def test_full_jitter_bounds():
    """full jitter can wait no time at all, and never the whole window"""
    assert backoff_delay(4, 100, 10_000, lambda: 0.0) == 0
    assert 1_599 < backoff_delay(4, 100, 10_000, lambda: 0.9999) < 1_600


def test_cap():
    """the window never grows past the cap"""
    assert backoff_delay(7, 100, 10_000, lambda: 0.5) == 5_000
    assert backoff_delay(20, 100, 10_000, lambda: 0.5) == 5_000
    assert backoff_delay(6, 100, 6_400, lambda: 0.5) == 3_200


def test_retry_delays():
    """one delay before each retry, from retry 0 upwards"""
    assert retry_delays(4, 100, 10_000, lambda: 0.5) == [50, 100, 200]
    assert retry_delays(1, 100, 10_000, lambda: 0.5) == []
    assert retry_delays(4, 100, 150, lambda: 0.25) == [25, 37.5, 37.5]


def test_retryable_failures():
    """timeouts and the statuses that can succeed later are retried when the call is safe to repeat"""
    assert should_retry(None, "GET") is True
    assert should_retry(503, "GET") is True
    assert should_retry(429, "PUT") is True
    assert should_retry(504, "DELETE") is True


def test_permanent_failures():
    """client errors other than 408 and 429 will fail again, so they are never retried"""
    assert should_retry(400, "GET") is False
    assert should_retry(404, "GET") is False
    assert should_retry(422, "PUT") is False
    assert should_retry(408, "GET") is True


def test_unsafe_methods():
    """a POST is repeated only with an idempotency key"""
    assert should_retry(None, "POST") is False
    assert should_retry(503, "POST") is False
    assert should_retry(503, "POST", has_idempotency_key=True) is True
    assert should_retry(400, "POST", has_idempotency_key=True) is False

Interview questions

Warm-up (fresher to mid level): why add jitter to exponential backoff? Without jitter, clients that failed at the same moment compute the same delays and retry at the same moment, so every retry arrives as one wave and the overloaded server rejects most of it again. Jitter makes each client wait a random part of its backoff window, which spreads the retries over time: the server sees a steady trickle it can serve instead of synchronised spikes. Full jitter, a random wait between zero and the capped exponential delay, is the common default. The window must be wide enough for the crowd: 1,000 clients against a server that takes 5 requests a millisecond need at least 200 ms.

Key takeaways

• Every remote call needs a timeout; library defaults often have none. Pick it from the dependency’s measured latency, for example the p99.9 if you accept cutting off 0.1 % of good calls.
• A timeout bounds each wait, not how many waits pile up: during a hang, calls in flight equal the call rate times the timeout, so a slow dependency also needs a cap on concurrent calls.
• Set a deadline once at the edge and pass the time left down the chain; drop work whose deadline has passed.
• Retry only failures that can succeed later, and only requests that are safe to repeat: idempotent methods, or calls with an idempotency key. Respect Retry-After.
• Use capped exponential backoff with jitter. Without jitter, clients that fail together retry together, and in the simulation none of 1,000 clients was ever served.
• Retries multiply across layers (attempts per layer to the power of the layers): retry at one layer and return “overloaded, do not retry” upwards.
• A retry budget, such as retries below 10 % of requests, holds an outage’s extra load near 1.1 times, at some cost to success during partial failures.