Design Patterns (GoF and modern) Module 8 – Resilience patterns as code

Retry with exponential backoff and jitter

Write a retry policy as code: capped exponential backoff with jitter, retryable errors and Retry-After, retry budgets, and tests with a fake clock.

Intermediate
30 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0, quickjs 0.32.0 and Go 1.27.1
By MySmartCoPilot

Free: the whole lesson. With a pass: the model answers to its 2 interview questions.

What you will learn

Implement a retry policy with capped exponential backoff and jitter
Decide which failures are retryable and honour Retry-After
Prevent retry storms with one retry point, a retry budget and idempotent requests
Test retry logic deterministically with a fake clock and seeded randomness

Before you start

Strategy: swap algorithms at runtime

What you’ll learn · Before you start

What you will learn

Implement a retry policy with capped exponential backoff and jitter
Decide which failures are retryable and honour Retry-After
Prevent retry storms with one retry point, a retry budget and idempotent requests
Test retry logic deterministically with a fake clock and seeded randomness

Before you start

Strategy: swap algorithms at runtime

On this page

A retry policy calls an operation again when it fails for a reason that may not last, such as a timeout or a 503 from a server that is busy or restarting. Exponential backoff makes each wait before a retry twice as long as the one before, up to a cap, so a struggling service gets more room with every failure. Jitter turns each wait into a random time inside that limit, so clients that failed at the same moment do not all come back at the same moment. In code, the policy is a decorator: it offers the same interface as the call it wraps, and it holds three smaller decisions, each a strategy object: which failures to retry, how long to wait, and whether the retry budget allows another try. The clock and the random source are passed in, so a test can check every wait exactly without waiting at all.

Every example on this page was executed with Python 3.14.8, Node.js 24.21.0 or Go 1.27.1, and the Python and JavaScript programs also start right here on the page.

A retry decorator around a service that fails twice

The shipping service in the example answers 503 Service Unavailable twice and then gives a quote. The caller calls fetch_quote() once; the decorator in front of the service does the rest:

A retry policy is a decorator that holds three smaller decisions

Text description of the diagram

The diagram shows one call passing through a retry decorator, from top to bottom.

The caller calls fetch_quote(order) once. The object it calls is the retry decorator, which has the same interface as the shipping service, so the caller cannot tell them apart.
The decorator sends attempt 1, 2, 3 and so on to the shipping service, which fails or answers each time.
After a failure, three questions follow, each answered by a smaller object: should this failure be retried, is there a retry left in the budget, and how long should the decorator wait.
The waiting goes through a clock and a random source that are passed in from outside: real ones in production, fake ones in tests, so a test checks exact waits without waiting.

A retry decorator with full jitter, in Python and JavaScript

Python · policy/retry.py

"""A retry policy written as a decorator, with its backoff as a strategy object.

The quote service below fails twice with 503 Service Unavailable and then answers. The decorator waits between
attempts with capped exponential backoff and full jitter. Time and randomness come from outside: a fake clock that
moves only when the policy sleeps, and a seeded generator, so the waits are the same on every run. The JavaScript
and Go versions use the same generator and print the same lines.
"""
import functools


def mulberry32(seed):
    """A small seeded generator of floats in [0, 1), written so that every language gets the same numbers."""
    state = seed & 0xFFFFFFFF

    def rand():
        nonlocal state
        state = (state + 0x6D2B79F5) & 0xFFFFFFFF
        t = ((state ^ (state >> 15)) * (state | 1)) & 0xFFFFFFFF
        t = ((t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF) ^ t
        return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296

    return rand


class FakeClock:
    """Milliseconds that pass only when someone sleeps."""

    def __init__(self):
        self.now_ms = 0

    def sleep(self, ms):
        self.now_ms += ms


class FullJitter:
    """The backoff strategy: a random wait between 0 and min(cap, base * 2 ** (retry - 1)) milliseconds."""

    def __init__(self, base_ms, cap_ms, rand):
        self.base_ms = base_ms
        self.cap_ms = cap_ms
        self.rand = rand

    def window_ms(self, retry):
        return min(self.cap_ms, self.base_ms * 2 ** (retry - 1))

    def delay_ms(self, retry):
        return int(self.rand() * self.window_ms(retry))


class Unavailable(Exception):
    """A failure worth retrying: the server is busy or restarting."""


def retrying(max_attempts, backoff, sleep, retry_on=(Unavailable,), log=print):
    """Decorates a function so that a failure listed in retry_on is retried, up to max_attempts calls in all."""

    def decorate(call):
        @functools.wraps(call)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return call(*args, **kwargs)
                except retry_on:
                    if attempt == max_attempts:
                        raise  # out of attempts: the caller sees the last error
                    wait = backoff.delay_ms(attempt)
                    log(f"  wait {wait} ms (window {backoff.window_ms(attempt)} ms)")
                    sleep(wait)

        return wrapper

    return decorate


def main():
    clock = FakeClock()
    replies = ["503 Service Unavailable", "503 Service Unavailable", "200 OK"]

    @retrying(max_attempts=4, backoff=FullJitter(base_ms=100, cap_ms=1000, rand=mulberry32(11)), sleep=clock.sleep)
    def fetch_quote(order):
        """Asks the shipping service how much delivering an order costs."""
        reply = replies.pop(0)
        print(f"call at {clock.now_ms} ms: {reply}")
        if reply.startswith("503"):
            raise Unavailable(reply)
        return f"{order}: delivery 4,900 paise"

    print(fetch_quote("order-41"))
    print(f"the decorated function is still called {fetch_quote.__name__}")


if __name__ == "__main__":
    main()

Output

call at 0 ms: 503 Service Unavailable
  wait 51 ms (window 100 ms)
call at 51 ms: 503 Service Unavailable
  wait 105 ms (window 200 ms)
call at 156 ms: 200 OK
order-41: delivery 4,900 paise
the decorated function is still called fetch_quote

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 retry.py

JavaScript · policy/retry.mjs

// A retry policy written as a decorator, with its backoff as a strategy object: the same program as retry.py.
// The quote service fails twice with 503 Service Unavailable and then answers. Time comes from a fake clock and
// randomness from the same seeded generator as the Python and Go versions, so all three print the same waits.

/** A small seeded generator of floats in [0, 1), written so that every language gets the same numbers. */
function mulberry32(seed) {
  let state = seed | 0;
  return () => {
    state = (state + 0x6d2b79f5) | 0;
    let t = Math.imul(state ^ (state >>> 15), state | 1);
    t = (t + Math.imul(t ^ (t >>> 7), t | 61)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

/** Milliseconds that pass only when someone sleeps. */
class FakeClock {
  nowMs = 0;
  sleep = (ms) => {
    this.nowMs += ms;
  };
}

/** The backoff strategy: a random wait between 0 and min(cap, base * 2 ** (retry - 1)) milliseconds. */
class FullJitter {
  constructor(baseMs, capMs, rand) {
    Object.assign(this, { baseMs, capMs, rand });
  }
  windowMs(retry) {
    return Math.min(this.capMs, this.baseMs * 2 ** (retry - 1));
  }
  delayMs(retry) {
    return Math.floor(this.rand() * this.windowMs(retry));
  }
}

/** A failure worth retrying: the server is busy or restarting. */
class Unavailable extends Error {}

/** Decorates a function so that a failure that retryOn accepts is retried, up to maxAttempts calls in all. */
function retrying({ maxAttempts, backoff, sleep, retryOn = (e) => e instanceof Unavailable }) {
  return (call) => {
    const wrapper = (...args) => {
      for (let attempt = 1; ; attempt++) {
        try {
          return call(...args);
        } catch (error) {
          if (!retryOn(error) || attempt === maxAttempts) throw error; // the caller sees the last error
          const wait = backoff.delayMs(attempt);
          console.log(`  wait ${wait} ms (window ${backoff.windowMs(attempt)} ms)`);
          sleep(wait);
        }
      }
    };
    Object.defineProperty(wrapper, 'name', { value: call.name }); // what functools.wraps does in Python
    return wrapper;
  };
}

const clock = new FakeClock();
const replies = ['503 Service Unavailable', '503 Service Unavailable', '200 OK'];

/** Asks the shipping service how much delivering an order costs. */
function fetchQuote(order) {
  const reply = replies.shift();
  console.log(`call at ${clock.nowMs} ms: ${reply}`);
  if (reply.startsWith('503')) throw new Unavailable(reply);
  return `${order}: delivery 4,900 paise`;
}

const quote = retrying({ maxAttempts: 4, backoff: new FullJitter(100, 1000, mulberry32(11)), sleep: clock.sleep })(fetchQuote);
console.log(quote('order-41'));
console.log(`the decorated function is still called ${quote.name}`);

Output

call at 0 ms: 503 Service Unavailable
  wait 51 ms (window 100 ms)
call at 51 ms: 503 Service Unavailable
  wait 105 ms (window 200 ms)
call at 156 ms: 200 OK
order-41: delivery 4,900 paise
the decorated function is still called fetchQuote

Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node retry.mjs

The first retry waited 51 ms of a 100 ms window and the second 105 ms of a 200 ms window, so the third call went out at 156 ms and got the quote. Both programs print the same waits because both use the same small generator, seeded with 11: the waits are random to the service, but entirely predictable to whoever chose the seed. The fake clock moves only when the policy sleeps, so the whole run takes no real time.

Two details keep the decorator honest. In Python, functools.wraps copies the wrapped function’s name and docstring onto the wrapper, which is why the last line still says fetch_quote; the JavaScript version sets the name by hand. And after the last attempt the decorator re-raises the error it caught, so the caller sees what actually went wrong instead of a vague “retries exhausted”.

The same design in Go is a function that takes a Fetch and returns another Fetch. The backoff is an interface, and errors.Is decides what to retry by looking through wrapped errors for ErrUnavailable:

retry.go: the decorator and the backoff strategy in Go Go · go/retry.go

// A retry policy as a decorator in Go: Retrying wraps any function of the Fetch type and returns another Fetch.
// The backoff is a strategy behind the Backoff interface, and the clock and the random generator are passed in,
// so this program prints the same waits as retry.py and retry.mjs.
package main

import (
	"errors"
	"fmt"
	"strings"
)

// Mulberry32 is the same small seeded generator as in the Python and JavaScript versions.
type Mulberry32 struct{ state uint32 }

func (m *Mulberry32) Float() float64 {
	m.state += 0x6D2B79F5
	t := m.state
	t = (t ^ (t >> 15)) * (t | 1)
	t ^= t + (t^(t>>7))*(t|61)
	return float64(t^(t>>14)) / 4294967296
}

// FakeClock holds milliseconds that pass only when someone sleeps.
type FakeClock struct{ NowMs int }

func (c *FakeClock) Sleep(ms int) { c.NowMs += ms }

// Backoff is the strategy that says how long to wait before retry number n (1, 2, 3 ...).
type Backoff interface {
	WindowMs(retry int) int
	DelayMs(retry int) int
}

// FullJitter waits a random time between 0 and min(cap, base * 2^(retry-1)) milliseconds.
type FullJitter struct {
	BaseMs, CapMs int
	Rand          *Mulberry32
}

func (b FullJitter) WindowMs(retry int) int { return min(b.CapMs, b.BaseMs<<(retry-1)) }
func (b FullJitter) DelayMs(retry int) int  { return int(b.Rand.Float() * float64(b.WindowMs(retry))) }

// ErrUnavailable marks a failure worth retrying: the server is busy or restarting.
var ErrUnavailable = errors.New("503 Service Unavailable")

// Fetch is the interface that both the service and its decorator implement.
type Fetch func(order string) (string, error)

// Retrying decorates call: a failure that wraps ErrUnavailable is retried, up to maxAttempts calls in all.
func Retrying(call Fetch, maxAttempts int, backoff Backoff, sleep func(int)) Fetch {
	return func(order string) (string, error) {
		for attempt := 1; ; attempt++ {
			reply, err := call(order)
			if err == nil || !errors.Is(err, ErrUnavailable) || attempt == maxAttempts {
				return reply, err // success, an error not worth retrying, or the last attempt's error
			}
			wait := backoff.DelayMs(attempt)
			fmt.Printf("  wait %d ms (window %d ms)\n", wait, backoff.WindowMs(attempt))
			sleep(wait)
		}
	}
}

func main() {
	clock := &FakeClock{}
	replies := []string{"503 Service Unavailable", "503 Service Unavailable", "200 OK"}
	fetchQuote := func(order string) (string, error) {
		reply := replies[0]
		replies = replies[1:]
		fmt.Printf("call at %d ms: %s\n", clock.NowMs, reply)
		if strings.HasPrefix(reply, "503") {
			return "", fmt.Errorf("fetch quote: %w", ErrUnavailable)
		}
		return order + ": delivery 4,900 paise", nil
	}
	quote := Retrying(fetchQuote, 4, FullJitter{BaseMs: 100, CapMs: 1000, Rand: &Mulberry32{state: 11}}, clock.Sleep)
	reply, err := quote("order-41")
	if err != nil {
		fmt.Println("gave up:", err)
		return
	}
	fmt.Println(reply)
}

retry.go, run with Go 1.27.1 Python · go/go_session.py

"""Builds and runs retry.go with Go 1.27.1, printing each command after "$ " and what it wrote.

The Go toolchain comes from the same mise installation as the Python running this file. Everything Go writes (its
build cache, its module cache and the home folder) goes into this folder, Go never downloads another toolchain, and
a command that fails stops the session with exit status 1, so a failed build is never recorded as a result.
"""
import os
import subprocess
import sys
from pathlib import Path

INSTALLS = Path(sys.executable).resolve().parents[3]  # .../installs/python/3.14.8/bin/python3 -> .../installs
GO_BIN = INSTALLS / "go" / "1.27.1" / "bin"


def run(command):
    here = Path.cwd()
    env = dict(os.environ)
    for name, folder in (("HOME", "home"), ("TMPDIR", "tmp"), ("GOCACHE", "go-cache"), ("GOPATH", "go-path")):
        (here / folder).mkdir(exist_ok=True)
        env[name] = str(here / folder)
    env.update(PATH=f"{GO_BIN}{os.pathsep}/usr/bin{os.pathsep}/bin", GOTOOLCHAIN="local", GOTELEMETRY="off")
    print(f"$ {command}")
    result = subprocess.run(command, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
    print(result.stdout, end="")
    if result.returncode != 0:
        print(f"[exit status {result.returncode}]")
        sys.exit(1)


run("go version")
run("go run retry.go")

Output

$ go version
go version go1.27.1 darwin/arm64
$ go run retry.go
call at 0 ms: 503 Service Unavailable
  wait 51 ms (window 100 ms)
call at 51 ms: 503 Service Unavailable
  wait 105 ms (window 200 ms)
call at 156 ms: 200 OK
order-41: delivery 4,900 paise

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 go_session.py

The Go program waited 51 and 105 ms as well. Writing the policy as a decorator pays off here: the service knows nothing about retries, the caller knows nothing about them either, and the same decorator can wrap any other call with the same shape.

Five ways to wait, measured on 10,000 clients

The backoff strategy has one job: given the number of the retry, return a wait. Five strategies are common. A constant wait is the same every time. Exponential backoff doubles the wait after every failure, up to the cap. Marc Brooker’s article on the AWS Architecture Blog compares three ways to add jitter to it. Full jitter waits a random time between zero and the exponential value; equal jitter keeps half of that value and makes only the other half random; decorrelated jitter draws each wait between the base and three times the previous wait, capped. The program below gives each strategy to 10,000 simulated clients that all failed at the same moment, and counts how many retries land in the busiest 10 ms slice:

Five backoff strategies, one client and then 10,000 Python · schedules/schedules.py

"""Five backoff strategies for the same retry policy, compared on 10,000 clients that failed at the same moment.

Each strategy is an object with one method, delay_ms(retry, previous_ms), so the policy that uses it never changes:
that is the Strategy pattern. Base 100 ms, cap 1,000 ms, five retries after the first attempt. The clients are
simulated, every attempt fails, and an attempt takes no time, so the numbers show only what the waits do.
"""
import random
from collections import Counter

BASE_MS, CAP_MS, RETRIES, CLIENTS = 100, 1000, 5, 10_000


def capped(retry):
    return min(CAP_MS, BASE_MS * 2 ** (retry - 1))


class Constant:
    name = "constant 100 ms"

    def delay_ms(self, retry, previous_ms, rng):
        return BASE_MS


class Exponential:
    name = "exponential, no jitter"

    def delay_ms(self, retry, previous_ms, rng):
        return capped(retry)


class FullJitter:
    name = "full jitter"

    def delay_ms(self, retry, previous_ms, rng):
        return rng.random() * capped(retry)


class EqualJitter:
    name = "equal jitter"

    def delay_ms(self, retry, previous_ms, rng):
        half = capped(retry) / 2
        return half + rng.random() * half


class DecorrelatedJitter:
    name = "decorrelated jitter"

    def delay_ms(self, retry, previous_ms, rng):
        return min(CAP_MS, BASE_MS + rng.random() * (previous_ms * 3 - BASE_MS))


def schedule(strategy, rng):
    """The waits of one client, in milliseconds."""
    waits, previous = [], BASE_MS
    for retry in range(1, RETRIES + 1):
        previous = strategy.delay_ms(retry, previous, rng)
        waits.append(previous)
    return waits


strategies = [Constant(), Exponential(), FullJitter(), EqualJitter(), DecorrelatedJitter()]

print("one client's waits in ms (seed 11):")
for s in strategies:
    print(f"  {s.name:<23}", " ".join(f"{w:6,.0f}" for w in schedule(s, random.Random(11))))

print(f"\n{CLIENTS:,} clients, all failing at 0 ms:")
print(f"  {'strategy':<23} {'mean wait':>9} {'longest':>9} {'busiest 10 ms':>14}")
for s in strategies:
    rng = random.Random(2026)
    totals, slots = [], Counter()
    for _ in range(CLIENTS):
        at = 0.0
        for wait in schedule(s, rng):
            at += wait
            slots[int(at // 10)] += 1  # which 10 ms slice this retry lands in
        totals.append(at)
    mean, longest, busiest = sum(totals) / CLIENTS, max(totals), max(slots.values())
    print(f"  {s.name:<23} {mean:6,.0f} ms {longest:6,.0f} ms {busiest:8,} calls")

Output

one client's waits in ms (seed 11):
  constant 100 ms            100    100    100    100    100
  exponential, no jitter     100    200    400    800  1,000
  full jitter                 45    112    370    373    508
  equal jitter                73    156    385    586    754
  decorrelated jitter        190    364  1,000  1,000  1,000

10,000 clients, all failing at 0 ms:
  strategy                mean wait   longest  busiest 10 ms
  constant 100 ms            500 ms    500 ms   10,000 calls
  exponential, no jitter   2,500 ms  2,500 ms   10,000 calls
  full jitter              1,241 ms  2,357 ms    1,461 calls
  equal jitter             1,870 ms  2,429 ms    2,048 calls
  decorrelated jitter      2,371 ms  4,182 ms      647 calls

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 schedules.py

Without jitter, all 10,000 clients sent each retry in the same 10 ms, whether the wait was constant or exponential: backoff on its own only moves the crowd, it does not spread it. Every form of jitter broke the crowd up, and the three differ in what they trade:

Exponential, no jitter: a mean wait of 2,500 ms, at most 2,500 ms, and all 10,000 retries in the busiest 10 ms.
Full jitter: a mean wait of 1,241 ms, at most 2,357 ms, and 1,461 retries in the busiest 10 ms.
Equal jitter: a mean wait of 1,870 ms, at most 2,429 ms, and 2,048 retries in the busiest 10 ms.
Decorrelated jitter: a mean wait of 2,371 ms, at most 4,182 ms, and 647 retries in the busiest 10 ms.

Full jitter halved the average wait and spread the retries well, which is why it is a sensible default. Equal jitter waited longer and spread less, because half of every wait is fixed. Decorrelated jitter spread the retries most, but its waits grow by up to three times each step, so its slowest client waited 4,182 ms against 2,500 ms for plain backoff. The longest wait matters when you choose a deadline: five retries with decorrelated jitter and this cap can take more than four seconds before the last attempt even starts.

Which failures to retry, and what Retry-After changes

A retry helps only if the failure may not happen again and repeating the request does no harm. The first condition is about the error. Timeouts, reset connections, 408 Request Timeout, 429 Too Many Requests and server errors such as 503 often clear up, while 400, 404 or 422 come back the same way every time. The standard resilience handler of .NET is a useful reference: it retries responses of 500 and above, 408 and 429, failed requests and attempt timeouts. It is a default someone chose, not a law of HTTP, and it retries every method unless you exclude the unsafe ones.

The second condition is about the request. RFC 9110 defines an idempotent method as one where many identical requests are intended to have the effect of a single one, which covers PUT, DELETE and the safe methods such as GET, but not POST; it also asks clients not to repeat a non-idempotent request by themselves unless they know the repeat is harmless. A timed-out POST that creates a payment may already have charged the card, so it is retried only with an idempotency key that lets the server recognise the repeat, as the idempotency lesson of this module shows.

Retry-After is the server’s own answer to “how long should I wait?”. RFC 9110 allows two forms, whole seconds or an HTTP date, and on a 503 the header estimates how long the outage will last; RFC 6585 allows it on a 429 as well. Treat it as the shortest acceptable wait, not as a suggestion, and compare it with the time the caller has left:

Ten failures, and what the policy decides for each Python · decide/decide.py

"""Which failures to retry, and for how long to wait: the policy's classifier and its reading of Retry-After.

decide() is the strategy that a retry policy asks after every failure. It looks at what failed (a status code, a
timeout or a reset connection), whether repeating the request is safe (the method, an idempotency key), what the
server asked for (Retry-After) and how much of the caller's deadline is left, 60 s here. The policy's own backoff
is 200 ms.
"""
from datetime import datetime, timedelta, timezone
from email.utils import format_datetime, parsedate_to_datetime

RETRYABLE = {408, 429, 500, 502, 503, 504}  # timeouts, rate limits and server errors
SAFE_TO_REPEAT = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE"}  # idempotent methods (RFC 9110, section 9.2.2)
RECEIVED = datetime(2030, 1, 1, 9, 0, 0, tzinfo=timezone.utc)  # when each reply arrived, fixed for the example
BACKOFF_MS = 200


def retry_after_ms(header):
    """Retry-After holds delay-seconds or an HTTP date (RFC 9110, section 10.2.3); None if absent or unreadable."""
    if header is None:
        return None
    if header.isdigit():
        return int(header) * 1000
    try:
        when = parsedate_to_datetime(header)
    except (TypeError, ValueError):
        return None
    return max(0, int((when - RECEIVED).total_seconds() * 1000))


def decide(method, outcome, has_key=False, header=None, remaining_ms=60_000):
    repeatable = method in SAFE_TO_REPEAT or has_key
    if outcome in ("timeout", "connection reset"):
        if not repeatable:
            return "give up: it may have worked, and a repeat is unsafe"
        return f"retry in {BACKOFF_MS} ms: safe to repeat"
    if outcome not in RETRYABLE:
        return "give up: it will fail the same way again"
    if not repeatable:
        return "give up: the server may have acted"
    asked = retry_after_ms(header)
    wait = max(BACKOFF_MS, asked or 0)  # Retry-After is a lower bound on the wait
    if wait >= remaining_ms:
        return f"give up now: asked to wait {wait:,} ms, {remaining_ms:,} ms left"
    source = "as Retry-After asks" if asked and asked > BACKOFF_MS else "the policy's backoff"
    return f"retry in {wait:,} ms, {source}"


in_30_s = format_datetime(RECEIVED + timedelta(seconds=30), usegmt=True)
cases = [
    ("GET", "timeout", {}),
    ("POST", "timeout", {}),
    ("POST", "timeout", {"has_key": True}),
    ("GET", 503, {"header": "2"}),
    ("GET", 429, {"header": in_30_s}),
    ("GET", 429, {"header": "3600"}),
    ("GET", 503, {"header": "soon"}),
    ("POST", 500, {}),
    ("GET", 404, {}),
    ("PUT", 400, {}),
]
for method, outcome, extra in cases:
    request = method + (" + key" if extra.get("has_key") else "")
    header = extra.get("header")
    shown = "" if header is None else (" Retry-After: a date, +30 s" if header == in_30_s else f" Retry-After: {header}")
    print(f"{request:<10} {str(outcome) + shown:<30} {decide(method, outcome, **extra)}")

Output

GET        timeout                        retry in 200 ms: safe to repeat
POST       timeout                        give up: it may have worked, and a repeat is unsafe
POST + key timeout                        retry in 200 ms: safe to repeat
GET        503 Retry-After: 2             retry in 2,000 ms, as Retry-After asks
GET        429 Retry-After: a date, +30 s retry in 30,000 ms, as Retry-After asks
GET        429 Retry-After: 3600          give up now: asked to wait 3,600,000 ms, 60,000 ms left
GET        503 Retry-After: soon          retry in 200 ms, the policy's backoff
POST       500                            give up: the server may have acted
GET        404                            give up: it will fail the same way again
PUT        400                            give up: it will fail the same way again

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 decide.py

The policy waited 2,000 ms when the server asked for 2 seconds, although its own backoff said 200 ms, and it read the date form as a wait of 30,000 ms counted from when the reply arrived. It gave up at once on the request for 3,600 seconds, because waiting an hour inside a 60-second deadline holds a connection for an answer nobody will read. A header it could not read, soon, was ignored in favour of the backoff. And the POST that failed with a 500 was not retried: the server may have done the work before it failed.

HTTP Status Codes Reference Look up what 408, 429, 502, 503 and 504 mean before you add one to a retry policy.

Retry storms: one retry point and a budget

Retries multiply when several layers each have their own policy. If a gateway, an orders service and a payments service each try the call below them 3 times, one user request can send 3 × 3 × 3 = 27 calls to the database during an outage, exactly when it can least take them. The program builds that chain from the same decorator and counts:

A call chain of three services, counted while the database is down Python · storm/storm.py

"""Retries multiply across layers: a call chain gateway -> orders -> payments -> database, counted.

Each service calls the next one through its own retry decorator. The database is down, so every call to it fails,
and the program counts the calls that reach it. Waits are left out (they change when calls arrive, not how many),
so the numbers are exact.
"""


class Down(Exception):
    """The database did not answer."""


class Database:
    def __init__(self):
        self.calls = 0

    def handle(self):
        self.calls += 1
        raise Down("database unavailable")


class RetryBudget:
    """A token bucket for retries: every request adds `ratio` of a token, every retry takes a whole one."""

    def __init__(self, ratio, reserve):
        self.ratio, self.reserve, self.tokens = ratio, reserve, reserve

    def on_request(self):
        self.tokens = min(self.reserve, self.tokens + self.ratio)

    def allow_retry(self):
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False


class Retrying:
    """The decorator: the same handle() as the service it wraps, with up to `attempts` tries."""

    def __init__(self, inner, attempts, budget=None):
        self.inner, self.attempts, self.budget = inner, attempts, budget

    def handle(self):
        if self.budget:
            self.budget.on_request()
        for attempt in range(1, self.attempts + 1):
            try:
                return self.inner.handle()
            except Down:
                last = attempt == self.attempts
                if last or (self.budget and not self.budget.allow_retry()):
                    raise


class Service:
    """A service that does its own work and then calls the next one."""

    def __init__(self, downstream):
        self.downstream = downstream

    def handle(self):
        return self.downstream.handle()


def chain(retrying_layers, budget=None):
    """gateway -> orders -> payments -> database; the named layers retry their downstream call 3 times."""
    db = Database()
    node = db
    for name in ("payments", "orders", "gateway"):
        if name in retrying_layers:
            node = Retrying(node, attempts=3, budget=budget if name == "payments" else None)
        node = Service(node)
    return node, db


def database_calls(requests, retrying_layers, budget=None):
    edge, db = chain(retrying_layers, budget)
    for _ in range(requests):
        try:
            edge.handle()
        except Down:
            pass  # the user sees an error, quickly or slowly
    return db.calls


every = ("gateway", "orders", "payments")
print("calls that reach the database while it is down:")
print(f"  {'':44}{'1 request':>10}{'1,000 requests':>16}")
for label, layers, budget in [
    ("every layer retries, 3 attempts each", every, None),
    ("only payments retries, 3 attempts", ("payments",), None),
    ("only payments retries, with a 10 % budget", ("payments",), "budget"),
]:
    one = database_calls(1, layers, RetryBudget(0.1, 10) if budget else None)
    many = database_calls(1000, layers, RetryBudget(0.1, 10) if budget else None)
    print(f"  {label:<44}{one:>10,}{many:>16,}")

Output

calls that reach the database while it is down:
                                               1 request  1,000 requests
  every layer retries, 3 attempts each                27          27,000
  only payments retries, 3 attempts                    3           3,000
  only payments retries, with a 10 % budget            3           1,109

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 storm.py

Moving the retries to one layer, the payments service next to the database, cut 27 calls per request to 3. The Amazon Builders’ Library works through a five-layer stack with three tries per layer, where the load on the database grows 243 times, and recommends retrying at a single point in the stack. The other layers still need to fail fast on the error they get, which is easiest when the error says that the lower layer has already retried.

A retry budget bounds the rest. The budget in the program is a token bucket: every request adds a tenth of a token, every retry spends a whole one, and the bucket holds at most 10. While the database was down, 1,000 requests sent 1,109 calls instead of 3,000, so the outage added about 11 % to its load instead of tripling it. When the database is healthy, retries are rare and the budget never runs out, so it changes nothing on a good day. AWS SDKs have limited retries with a local token bucket of this kind since 2016, according to the same article.

A request that crosses several services also needs a deadline that travels with it, so that the layer at the bottom stops retrying once the user has given up; the system-design lesson on timeouts, retries and backoff works through deadlines and budgets across a whole system.

Testing a retry policy without sleeping

A test that really sleeps is slow and still cannot check the waits, because they are random. Pass the policy a fake clock and a seeded generator instead, and every wait becomes a value the test can assert:

Four tests of the retry decorator, with a fake clock Python · policy/test_retry.py

"""Tests for the decorator of retry.py: exact waits for a seed, and not one millisecond of real waiting.

Each test builds the decorator with a fake clock and a seeded generator, wraps a scripted fake service and checks
when each call happened. The same tests run under pytest; here a few lines at the end run them and report.
"""
import time

from retry import FakeClock, FullJitter, Unavailable, mulberry32, retrying


def scripted(clock, replies, calls):
    """A fake service: it records when it was called and gives the next scripted reply (or raises it)."""

    def call():
        calls.append(clock.now_ms)
        reply = replies.pop(0)
        if isinstance(reply, Exception):
            raise reply
        return reply

    return call


def build(clock, replies, calls, attempts=4, rand=None):
    backoff = FullJitter(100, 1000, rand or mulberry32(11))
    quiet = lambda message: None  # noqa: E731 - the decorator's log is not part of what is tested
    return retrying(attempts, backoff, clock.sleep, log=quiet)(scripted(clock, replies, calls))


def test_waits_follow_the_seed():
    clock, calls = FakeClock(), []
    fetch = build(clock, [Unavailable(), Unavailable(), "200 OK"], calls)
    assert fetch() == "200 OK"
    assert calls == [0, 51, 156]


def test_gives_up_after_the_last_attempt():
    clock, calls = FakeClock(), []
    fetch = build(clock, [Unavailable()] * 4, calls)
    try:
        fetch()
    except Unavailable:
        pass
    else:
        raise AssertionError("the fourth failure should reach the caller")
    assert len(calls) == 4


def test_other_errors_are_not_retried():
    clock, calls = FakeClock(), []
    fetch = build(clock, [ValueError("bad order id"), "200 OK"], calls)
    try:
        fetch()
    except ValueError:
        pass
    assert calls == [0]


def test_no_wait_is_longer_than_the_cap():
    clock, calls = FakeClock(), []
    fetch = build(clock, [Unavailable()] * 9 + ["200 OK"], calls, attempts=10, rand=lambda: 0.999999)
    fetch()
    gaps = [b - a for a, b in zip(calls, calls[1:])]
    assert max(gaps) == 999, gaps


started = time.perf_counter()
tests = [test_waits_follow_the_seed, test_gives_up_after_the_last_attempt, test_other_errors_are_not_retried,
         test_no_wait_is_longer_than_the_cap]
for test in tests:
    test()
    print(f"ok   {test.__name__}")
assert time.perf_counter() - started < 1, "the tests waited for real"
print(f"{len(tests)} tests passed without sleeping")

Output

ok   test_waits_follow_the_seed
ok   test_gives_up_after_the_last_attempt
ok   test_other_errors_are_not_retried
ok   test_no_wait_is_longer_than_the_cap
4 tests passed without sleeping

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 test_retry.py

The first test checks that the calls happened at 0, 51 and 156 ms, the schedule of seed 11. The others check that the fourth failure reaches the caller, that a ValueError is not retried, and that no wait reaches the cap even when the generator returns its largest value. Nine retries of up to a second each ran without any real waiting. Tests like these catch the bugs that matter in retry code: an off-by-one in the attempt count, a window that keeps growing past the cap, an error that should never be retried.

Practise: write the policy

The exercise asks for the policy as a class: delay_ms() with full jitter and a cap, and run() that retries only TransientError and treats a Retry-After value as the shortest wait. The sample tests script the random source and record every sleep.

Exercise · Medium · Python

A retry policy with full jitter and Retry-After

Write the two methods of RetryPolicy in retry_policy.py. The policy is built with the most calls it may make, a base and a cap in milliseconds, a random source rand() that returns a float from 0 up to 1, and a sleep(ms) function. Both are passed in, so the tests can script them.

**delay_ms(retry)** returns the wait before retry number retry (1 for the first retry) with full jitter: the window is base_ms doubled once for every earlier retry and never more than cap_ms, and the wait is rand() times the window, with the fraction dropped.
**run(call)** calls call() and returns what it returns. When it raises TransientError, the policy sleeps and calls again, up to max_attempts calls in all; after the last one it lets the error reach the caller. If the error carries retry_after_ms, the server asked for at least that wait, so the policy sleeps for the larger of the two. Any other exception reaches the caller at once, without a retry.

The sample tests record every sleep() call, so they check the exact waits without waiting at all.

Starter code · retry_policy.py

"""A retry policy: capped exponential backoff with full jitter, and Retry-After as the shortest wait."""


class TransientError(Exception):
    """A failure worth retrying. retry_after_ms is the wait the server asked for, or None."""

    def __init__(self, message="", retry_after_ms=None):
        super().__init__(message)
        self.retry_after_ms = retry_after_ms


class RetryPolicy:
    def __init__(self, max_attempts, base_ms, cap_ms, rand, sleep):
        self.max_attempts = max_attempts
        self.base_ms = base_ms
        self.cap_ms = cap_ms
        self.rand = rand    # rand() returns a float from 0 up to (not including) 1
        self.sleep = sleep  # sleep(ms) waits, or moves a fake clock in the tests

    def delay_ms(self, retry):
        """The wait before retry number `retry` (1 for the first retry), in whole milliseconds."""
        raise NotImplementedError("write delay_ms")

    def run(self, call):
        """Calls call(); retries TransientError, up to max_attempts calls in all."""
        raise NotImplementedError("write run")

The sample tests · test_retry_policy.py

from retry_policy import RetryPolicy, TransientError


def scripted(values):
    """A random source that returns the given numbers in turn."""
    values = list(values)
    return lambda: values.pop(0)


def service(replies):
    """A fake service: each call takes the next reply and raises it if it is an exception."""
    calls = []

    def call():
        calls.append(len(calls) + 1)
        reply = replies.pop(0)
        if isinstance(reply, BaseException):
            raise reply
        return reply

    return call, calls


def test_success_at_once_never_sleeps():
    """returns the first answer without sleeping"""
    slept = []
    call, calls = service(["ok"])
    policy = RetryPolicy(4, 100, 1000, scripted([0.5]), slept.append)
    assert policy.run(call) == "ok"
    assert (calls, slept) == ([1], [])


def test_full_jitter_waits():
    """waits rand() times a window that doubles: 100, 200, then 400 ms"""
    policy = RetryPolicy(4, 100, 1000, scripted([0.5, 0.25, 0.75]), lambda ms: None)
    assert [policy.delay_ms(n) for n in (1, 2, 3)] == [50, 50, 300]


def test_the_cap_limits_every_window():
    """never waits longer than the cap, however many retries"""
    policy = RetryPolicy(10, 100, 250, lambda: 0.999, lambda ms: None)
    assert [policy.delay_ms(n) for n in (1, 2, 3, 4, 5)] == [99, 199, 249, 249, 249]


def test_retries_until_an_answer():
    """sleeps between failed attempts and returns the answer that finally comes"""
    slept = []
    call, calls = service([TransientError("503"), TransientError("503"), "ok"])
    policy = RetryPolicy(4, 100, 1000, scripted([0.5, 0.5]), slept.append)
    assert policy.run(call) == "ok"
    assert (calls, slept) == ([1, 2, 3], [50, 100])


def test_gives_up_after_the_last_attempt():
    """lets the last TransientError reach the caller after max_attempts calls"""
    slept = []
    call, calls = service([TransientError("503")] * 3)
    policy = RetryPolicy(3, 100, 1000, scripted([0.5, 0.5]), slept.append)
    try:
        policy.run(call)
    except TransientError:
        pass
    else:
        raise AssertionError("expected the last TransientError")
    assert (calls, slept) == ([1, 2, 3], [50, 100])


def test_other_errors_are_not_retried():
    """raises any other error at once, without sleeping"""
    slept = []
    call, calls = service([ValueError("bad order id"), "ok"])
    policy = RetryPolicy(4, 100, 1000, scripted([0.5]), slept.append)
    try:
        policy.run(call)
    except ValueError:
        pass
    else:
        raise AssertionError("expected the ValueError")
    assert (calls, slept) == ([1], [])


def test_retry_after_is_the_shortest_wait():
    """waits at least as long as the server asked, and the backoff when that is longer"""
    slept = []
    call, calls = service([TransientError("429", retry_after_ms=500), TransientError("503", retry_after_ms=10), "ok"])
    policy = RetryPolicy(4, 100, 1000, scripted([0.5, 0.5]), slept.append)
    assert policy.run(call) == "ok"
    assert slept == [500, 100]

A hint

Start with delay_ms: the window for retry n is min(cap_ms, base_ms * 2 ** (n - 1)), and int(rand() * window) drops the fraction. In run, loop over the attempts from 1 to max_attempts and put return call() inside a try; in the except TransientError branch, re-raise on the last attempt, otherwise work out the wait, raise it to retry_after_ms when the error has one, and sleep.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Python Online Compiler Run schedules.py with your own base, cap and number of retries, and see how the busiest slice changes.

Interview questions

Warm-up · Which failures should a retry policy retry, and which should it never retry?

Retry failures that may not happen again and whose repeat is safe. The first test is the kind of failure: timeouts, reset connections, 408 Request Timeout, 429 Too Many Requests and server errors such as 502, 503 and 504 often clear up, while 400, 401, 403, 404 and 422 come back the same way every time, so retrying them only adds load. The second test is the request: GET, PUT and DELETE are idempotent in HTTP, so sending them twice has the effect of sending them once, but a POST that timed out may already have charged a card, and repeating it is safe only with an idempotency key. Finally, treat Retry-After as the shortest wait, and give up when the wait would pass the caller’s deadline.

Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.

Choose a pass

Mid to senior

A mobile app, an API gateway and an orders service each retry a failing call three times. During an outage of the payments service below them, payments receives 27 times its normal load and cannot recover. How do you change the retries, and how do you show that the change works?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to senior

A partner's API answers your checkout service with 429 Too Many Requests and Retry-After: 3600. Your retry policy obeys the header, and checkout requests hang until users give up. What should the policy have done, and what would you change around it?

Model answer, Pro. First step and follow-ups

Key takeaways

A retry policy is a decorator: the same interface as the call it wraps, with strategies for what to retry, how long to wait and how many retries the budget allows.
Exponential backoff alone moves a crowd of clients but does not spread it; with jitter, the busiest 10 ms of 10,000 clients fell from 10,000 retries to between 647 and 2,048.
Full jitter is a sensible default: it halved the average wait. Decorrelated jitter spreads most but can wait much longer, so size deadlines from the longest wait.
Retry only failures that may pass and requests that are safe to repeat, treat Retry-After as the shortest wait, and give up when the wait would pass the deadline.
Retry at one layer and give it a budget: 27 database calls per request fell to 3, and an outage’s extra load fell from 3 times to about 1.1 times.
Pass the clock and the random source in, and tests check exact waits without sleeping.

References

Timeouts, retries, and backoff with jitter (Amazon Builders' Library) (Amazon Web Services)
Exponential backoff and jitter (AWS Architecture Blog) (Amazon Web Services)
RFC 9110: HTTP Semantics (idempotent methods, Retry-After) (IETF)
RFC 6585: Additional HTTP Status Codes (429 Too Many Requests) (IETF)
Build resilient HTTP apps: key development patterns (.NET) (Microsoft)
Retry pattern (Azure Architecture Center) (Microsoft)
Python 3.14 documentation: functools.wraps (Python Software Foundation)
Python 3.14 documentation: email.utils (parsedate_to_datetime) (Python Software Foundation)
Go standard library: package errors (Is and wrapping) (The Go Authors)

Report a problem with this lesson

In this module: Resilience patterns as code

Retry with exponential backoff and jitter Free lesson, Pro answers: 30 minutes
Circuit breaker: fail fast, recover safely Free core, Pro depth: 8 of 30 minutes free
Timeouts, deadlines and cancellation Free intro, Pro: 6 of 30 minutes free
Idempotency keys, outbox and saga in code Free intro, Pro: 10 of 40 minutes free

Your country

Retry with exponential backoff and jitter

What you will learn

Before you start

What you will learn

Before you start

A retry decorator around a service that fails twice

Five ways to wait, measured on 10,000 clients

Which failures to retry, and what Retry-After changes

Retry storms: one retry point and a budget

Testing a retry policy without sleeping

Practise: write the policy

A retry policy with full jitter and Retry-After

Results of the sample tests

Check yourself

Interview questions

A mobile app, an API gateway and an orders service each retry a failing call three times. During an outage of the payments service below them, payments receives 27 times its normal load and cannot recover. How do you change the retries, and how do you show that the change works?

A partner's API answers your checkout service with 429 Too Many Requests and Retry-After: 3600. Your retry policy obeys the header, and checkout requests hang until users give up. What should the policy have done, and what would you change around it?

Key takeaways

References

Related tools