Web Scraping & Browser Automation (Python and JavaScript) Module 5 – Politeness, caching and robust crawls
Rate limits, 429 and Retry-After
Pace a collector below a site's limit with a token bucket or a jittered pause, obey 429 and Retry-After, and retry only failures that can succeed.
Free lesson, Pro answers: 25 minutes
Free: the whole lesson. With a pass: the model answers to its 2 interview questions.
What you will learn
- Limit requests to one host with a token bucket of your own or a fixed pause with jitter
- Calculate the wait that Retry-After asks for, in seconds or as an HTTP date counted from the answer's Date
- Schedule retries with exponential backoff, full jitter and a cap when an answer gives no Retry-After
- Separate temporary failures from permanent ones, and retry only safe requests within a budget
Before you start
What you’ll learn · Before you start
What you will learn
- Limit requests to one host with a token bucket of your own or a fixed pause with jitter
- Calculate the wait that Retry-After asks for, in seconds or as an HTTP date counted from the answer's Date
- Schedule retries with exponential backoff, full jitter and a cap when an answer gives no Retry-After
- Separate temporary failures from permanent ones, and retry only safe requests within a budget
Before you start
On this page
The practice shop’s price API answers 5 requests at once and then 1 a second. A collector that asks for its 20 pages as fast as it can gets 5 of them, and one that simply asks again after every refusal sends 301 requests for those 20 pages, 281 of them refused.
This lesson is about the client’s side of a rate limit. You will read 429 Too Many Requests and its Retry-After,
stay under a limit with a pause or a token bucket of your own, back off with jitter when the server names no time,
and decide which failures deserve a retry at all. Every measurement below comes from the lab shop’s own access log.
A rate limit is a budget, and 429 says it is spent
Send the price API twenty requests in a row, with no pause between them, and read what comes back. Each request takes 50 ms on the shop’s clock:
limits/burst.py
"""Twenty requests to the price API, one after another and with no pause: what its limit lets through."""
import nimbu_lab as lab
lab.reset()
answers = []
for page in range(1, 21):
status, headers = lab.get(f"/api/v1/prices?page={page}")
answers.append((page, status, headers["Retry-After"]))
print("page status Retry-After")
for page, status, retry_after in answers:
if page <= 6 or page == 20:
print(f"{page:>4} {status:>6} {retry_after or '-'}")
elif page == 7:
print(" … … …")
ok = sum(status == 200 for _, status, _ in answers)
print(f"{ok} answered and {len(answers) - ok} refused in {lab.clock.now():.2f} s on the shop's clock")Output
page status Retry-After 1 200 - 2 200 - 3 200 - 4 200 - 5 200 - 6 429 1 … … … 20 429 1 5 answered and 15 refused in 1.00 s on the shop's clock
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 burst.py
nimbu_lab.py (the shop) · limits/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, with the pages of the rate-limit lesson. Every product, price and
person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers, and one change: the last step hands each request to app() instead of opening
a socket, so nothing leaves your program. Time is the shop's clock. Each request takes 50 ms of it, and sleep() moves
it on instead of waiting, so a run that spans minutes ends at once and prints the same numbers every time.
Two hosts answer:
bazaar.nimbu.test the price API, /api/v1/prices?page=1 to 20: a token bucket per client, 5 requests at once and
1 more each second, then 429 Too Many Requests with Retry-After
flaky.nimbu.test /board (closed for maintenance for the first 90 s, 503 with Retry-After as an HTTP date),
/stock?item=n (serves 5 requests in each second and answers the rest 503), /report/<name>
(each name fails in its own way, see REPORTS) and /catalogue?page=n (503 to everything while
OUTAGE["on"] is True)
"""
import email.utils
import http.client
import io
import json
import math
import urllib.error
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
SHOP = "bazaar.nimbu.test"
FLAKY = "flaky.nimbu.test"
EPOCH = 1_800_000_000 # the clock reads 0 at this moment, in seconds since 1970 (what Date headers are made from)
LATENCY = 0.05 # seconds a request takes, there and back
class Clock:
"""The shop's clock, in seconds. sleep() moves it on instead of waiting."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += max(0.0, seconds)
clock = Clock()
LOG = [] # what the shop saw: one entry per request
SKEW = {"flaky": 0.0} # how far the flaky host's clock is from yours: -40.0 means 40 s behind
COUNTS = {} # attempts per /report/ path, so a page can fail twice and then work
OUTAGE = {"on": False} # True: the flaky host's catalogue is down
class Bucket:
"""The API's limit for one client: at most `capacity` tokens, `rate` new tokens a second, one per request."""
def __init__(self, capacity, rate, now):
self.capacity, self.rate, self.tokens, self.at = capacity, rate, float(capacity), now
def take(self, now):
"""(True, 0) when a token was taken; (False, seconds until the next one) when the bucket is empty."""
self.tokens = min(self.capacity, self.tokens + (now - self.at) * self.rate)
self.at = now
if self.tokens >= 1 - 1e-9:
self.tokens -= 1
return True, 0.0
return False, (1 - self.tokens) / self.rate
BUCKETS = {} # one Bucket per client, told apart by User-Agent
PER_SECOND = {} # /stock requests seen in each whole second
def reset(skew=0.0):
"""A fresh shop: clock at 0, empty log, full buckets; the flaky host's clock `skew` seconds from yours."""
clock.seconds = 0.0
for store in (LOG, BUCKETS, COUNTS, PER_SECOND):
store.clear()
SKEW["flaky"] = skew
OUTAGE["on"] = False
def http_date(seconds):
"""An HTTP date for a moment on the shop's clock, as Date and Retry-After carry it."""
return email.utils.formatdate(EPOCH + seconds, usegmt=True)
def answer(status, body="", kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def prices(query, who, now):
page = int(query.get("page", "1"))
if not 1 <= page <= 20:
return answer("404 Not Found", "no such page\n")
bucket = BUCKETS.setdefault(who, Bucket(5, 1.0, now))
ok, wait = bucket.take(now)
if not ok: # RFC 6585: 429 may say how long to wait; delay-seconds is a whole number
return answer("429 Too Many Requests", json.dumps({"error": "5 requests at once, then 1 a second"}),
"application/json", [("Retry-After", str(math.ceil(wait)))])
rows = [{"item": f"NB-{(page - 1) * 3 + i + 1:04d}", "paise": 2500 + 75 * ((page * 3 + i) % 40)} for i in range(3)]
return answer("200 OK", json.dumps({"page": page, "prices": rows}), "application/json")
REPORTS = { # /report/<name>: the answers each one gives, attempt after attempt (the last one repeats)
"daily": ["200 OK"],
"weekly": ["503 Service Unavailable", "503 Service Unavailable", "200 OK"],
"monthly": ["429 Too Many Requests", "200 OK"],
"archive": ["404 Not Found"],
"old": ["410 Gone"],
"staff": ["403 Forbidden"],
"broken": ["500 Internal Server Error"],
"gateway": ["504 Gateway Timeout", "200 OK"],
"bad-query": ["400 Bad Request"],
}
def flaky(method, path, query, now):
here = now + SKEW["flaky"] # what the flaky host's own clock says
dated = [("Date", http_date(here))]
if path == "/board":
if now < 90: # maintenance: back at 90 s on your clock, which its own clock calls 90 + skew
return answer("503 Service Unavailable", "the price board is being rebuilt\n",
extra=[*dated, ("Retry-After", http_date(90 + SKEW["flaky"]))])
return answer("200 OK", "price board: 40 rows\n", extra=dated)
if path == "/stock":
second = int(now)
PER_SECOND[second] = PER_SECOND.get(second, 0) + 1
if PER_SECOND[second] > 5: # overloaded: no Retry-After, because it cannot know when it will recover
return answer("503 Service Unavailable", "too busy\n", extra=dated)
return answer("200 OK", f"stock of item {query.get('item', '?')}: 12\n", extra=dated)
if path == "/catalogue":
if OUTAGE["on"]:
return answer("503 Service Unavailable", "the catalogue is down\n", extra=dated)
return answer("200 OK", f"catalogue page {query.get('page', '1')}\n", extra=dated)
if path == "/orders" and method == "POST":
return answer("503 Service Unavailable", "orders are paused\n", extra=dated)
name = path.removeprefix("/report/")
if path.startswith("/report/") and name in REPORTS:
n = COUNTS[path] = COUNTS.get(path, 0) + 1
status = REPORTS[name][min(n, len(REPORTS[name])) - 1]
extra = [("Retry-After", "2")] if status.startswith("429") else []
return answer(status, f"{name} report\n" if status.startswith("200") else "", extra=[*dated, *extra])
return answer("404 Not Found", "no such page\n", extra=dated)
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
now = clock.now()
host, method, path = environ.get("HTTP_HOST", ""), environ["REQUEST_METHOD"], environ["PATH_INFO"]
query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
who = environ.get("HTTP_USER_AGENT", "")
if host == SHOP and path == "/api/v1/prices":
status, headers, body = prices(query, who, now)
elif host == FLAKY:
status, headers, body = flaky(method, path, query, now)
elif host == SHOP:
status, headers, body = answer("404 Not Found", "no such page\n")
else:
status, headers, body = answer("421 Misdirected Request", "not a Nimbu Bazaar address\n")
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"t": round(now, 3), "host": host, "method": method, "target": target, "status": int(status[:3]),
"retry_after": dict(headers).get("Retry-After"), "who": who})
start_response(status, headers)
return [body]
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data)}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in headers.items()})
got = {}
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
clock.sleep(LATENCY) # the answer arrives 50 ms after the request left
raw = "".join(f"{k}: {v}\r\n" for k, v in got["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, int(got["status"][:3]))
response.reason = response.msg = got["status"][4:]
return response
http_open = https_open
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network."""
director = urllib.request.OpenerDirector()
for handler in [ShopTransport(), urllib.request.HTTPDefaultErrorHandler(), urllib.request.HTTPRedirectHandler(),
urllib.request.HTTPErrorProcessor(), urllib.request.UnknownHandler(), *handlers]:
director.add_handler(handler)
return director
def get(path, host=SHOP, who="NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)", method="GET"):
"""One request through urllib: (status, headers). urllib raises HTTPError for 4xx and 5xx; this returns them."""
request = urllib.request.Request(f"https://{host}{path}", headers={"User-Agent": who}, method=method,
data=b"{}" if method == "POST" else None)
try:
with OPENER.open(request, timeout=10) as response:
response.read()
return response.status, response.headers
except urllib.error.HTTPError as error:
error.read()
return error.code, error.headers
OPENER = opener()Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Five answers came back with data and fifteen with 429 and Retry-After: 1. RFC 6585 created 429 to tell a client
it “has sent too many requests in a given amount of time”, and the answer may name, in Retry-After, how long to
wait. The standard leaves the counting to each server: per resource, across a site or across a group of servers,
with the client told apart by its credentials or a cookie. Every API has a rate limit: The number of requests an API allows a client in a time window, such as 100 an hour. Answers often report how many are left, and requests beyond it are refused, usually with 429. More in “Use the API first: JSON, keys and quotas”
of its own, so read its documentation and its answers rather than guessing.
The shop counts with a token bucket: A way to limit a rate: a bucket holds up to a set number of tokens, each request spends one, and tokens come back at a steady rate, so a short burst passes and then a steady pace. per client. The bucket holds 5 tokens, every request spends one, and one token comes back each second, so a burst of 5 passes and after that 1 a second does. That is why requests 1 to 5 succeeded within 0.2 s and the next fifteen, sent before a token came back, did not. Try the same bucket yourself:
Send requests one at a time or in a burst, let time pass, and change the bucket size and the refill rate. A request needs one token; without one it is rejected with 429.
At the start: The bucket holds 5 of 5 tokens and gains 1 token a second, so up to 5 requests pass at once and 1 a second keep passing after that. No requests yet.
Send a burst of ten and watch five pass; wait two seconds and two more pass. Over a minute that starts with a full bucket, at most 65 requests can succeed: the 5 of the burst and one for each of the 60 seconds. Asking faster does not raise that number, it only adds refusals.
Not every limit says 429
RFC 6585 does not require a server to answer every excess request with 429: under a flood it may simply drop connections. A burst of timeouts or reset connections from a host you are pressing hard means the same as 429: slow down.
Stay under the limit: a pause with jitter, or a bucket of your own
A refusal costs you a request and costs the server work. The next program fetches the same 20 pages four ways, each until every page has arrived, and counts what reached the shop:
limits/pace.py
"""The same 20 pages fetched four ways: retrying at once, obeying Retry-After, a pause with jitter, a token bucket.
Every page must arrive in the end; the shop's log counts what each way cost it."""
import random
import nimbu_lab as lab
class TokenBucket:
"""The collector's own limit: `burst` requests may start at once, then `rate` a second."""
def __init__(self, rate, burst, now):
self.rate, self.burst, self.tokens, self.at = rate, burst, float(burst), now
def wait(self, now):
"""Seconds to wait before the next request may start (0 if a token is there); the token is then spent."""
self.tokens = min(self.burst, self.tokens + (now - self.at) * self.rate)
self.at = now
wait = 0.0 if self.tokens >= 1 else (1 - self.tokens) / self.rate
self.tokens += wait * self.rate - 1 # the tokens earned while waiting, minus the one this request spends
self.at = now + wait
return wait
def fetch_all(before_each=None, obey_retry_after=False):
lab.reset()
for page in range(1, 21):
while True:
if before_each:
lab.clock.sleep(before_each())
status, headers = lab.get(f"/api/v1/prices?page={page}")
if status == 200:
break
if obey_retry_after:
lab.clock.sleep(int(headers["Retry-After"]))
refused = sum(entry["status"] == 429 for entry in lab.LOG)
return len(lab.LOG), refused, lab.clock.now()
rng = random.Random(7) # seeded, so the jitter is the same on every run
pause, spread = 1.1, 0.10 # a pause of 1.1 s, varied by up to 10 % either way
bucket = TokenBucket(rate=0.9, burst=3, now=0.0)
ways = [
("ask again at once", fetch_all()),
("wait for Retry-After", fetch_all(obey_retry_after=True)),
(f"pause {pause:g} s, ±{spread * 100:g} % jitter",
fetch_all(before_each=lambda: rng.uniform(pause * (1 - spread), pause * (1 + spread)))),
(f"own bucket: {bucket.burst}, then {bucket.rate:g}/s",
fetch_all(before_each=lambda: bucket.wait(lab.clock.now()))),
]
print(f"{'way':<28}{'requests':>9}{'refused':>9}{'done at':>9}")
for name, (requests, refused, done) in ways:
print(f"{name:<28}{requests:>9}{refused:>9}{done:>8.1f}s")Output
way requests refused done at ask again at once 301 281 15.1s wait for Retry-After 34 14 15.7s pause 1.1 s, ±10 % jitter 20 0 22.4s own bucket: 3, then 0.9/s 20 0 18.9s
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pace.py
nimbu_lab.py (the shop) · limits/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, with the pages of the rate-limit lesson. Every product, price and
person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers, and one change: the last step hands each request to app() instead of opening
a socket, so nothing leaves your program. Time is the shop's clock. Each request takes 50 ms of it, and sleep() moves
it on instead of waiting, so a run that spans minutes ends at once and prints the same numbers every time.
Two hosts answer:
bazaar.nimbu.test the price API, /api/v1/prices?page=1 to 20: a token bucket per client, 5 requests at once and
1 more each second, then 429 Too Many Requests with Retry-After
flaky.nimbu.test /board (closed for maintenance for the first 90 s, 503 with Retry-After as an HTTP date),
/stock?item=n (serves 5 requests in each second and answers the rest 503), /report/<name>
(each name fails in its own way, see REPORTS) and /catalogue?page=n (503 to everything while
OUTAGE["on"] is True)
"""
import email.utils
import http.client
import io
import json
import math
import urllib.error
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
SHOP = "bazaar.nimbu.test"
FLAKY = "flaky.nimbu.test"
EPOCH = 1_800_000_000 # the clock reads 0 at this moment, in seconds since 1970 (what Date headers are made from)
LATENCY = 0.05 # seconds a request takes, there and back
class Clock:
"""The shop's clock, in seconds. sleep() moves it on instead of waiting."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += max(0.0, seconds)
clock = Clock()
LOG = [] # what the shop saw: one entry per request
SKEW = {"flaky": 0.0} # how far the flaky host's clock is from yours: -40.0 means 40 s behind
COUNTS = {} # attempts per /report/ path, so a page can fail twice and then work
OUTAGE = {"on": False} # True: the flaky host's catalogue is down
class Bucket:
"""The API's limit for one client: at most `capacity` tokens, `rate` new tokens a second, one per request."""
def __init__(self, capacity, rate, now):
self.capacity, self.rate, self.tokens, self.at = capacity, rate, float(capacity), now
def take(self, now):
"""(True, 0) when a token was taken; (False, seconds until the next one) when the bucket is empty."""
self.tokens = min(self.capacity, self.tokens + (now - self.at) * self.rate)
self.at = now
if self.tokens >= 1 - 1e-9:
self.tokens -= 1
return True, 0.0
return False, (1 - self.tokens) / self.rate
BUCKETS = {} # one Bucket per client, told apart by User-Agent
PER_SECOND = {} # /stock requests seen in each whole second
def reset(skew=0.0):
"""A fresh shop: clock at 0, empty log, full buckets; the flaky host's clock `skew` seconds from yours."""
clock.seconds = 0.0
for store in (LOG, BUCKETS, COUNTS, PER_SECOND):
store.clear()
SKEW["flaky"] = skew
OUTAGE["on"] = False
def http_date(seconds):
"""An HTTP date for a moment on the shop's clock, as Date and Retry-After carry it."""
return email.utils.formatdate(EPOCH + seconds, usegmt=True)
def answer(status, body="", kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def prices(query, who, now):
page = int(query.get("page", "1"))
if not 1 <= page <= 20:
return answer("404 Not Found", "no such page\n")
bucket = BUCKETS.setdefault(who, Bucket(5, 1.0, now))
ok, wait = bucket.take(now)
if not ok: # RFC 6585: 429 may say how long to wait; delay-seconds is a whole number
return answer("429 Too Many Requests", json.dumps({"error": "5 requests at once, then 1 a second"}),
"application/json", [("Retry-After", str(math.ceil(wait)))])
rows = [{"item": f"NB-{(page - 1) * 3 + i + 1:04d}", "paise": 2500 + 75 * ((page * 3 + i) % 40)} for i in range(3)]
return answer("200 OK", json.dumps({"page": page, "prices": rows}), "application/json")
REPORTS = { # /report/<name>: the answers each one gives, attempt after attempt (the last one repeats)
"daily": ["200 OK"],
"weekly": ["503 Service Unavailable", "503 Service Unavailable", "200 OK"],
"monthly": ["429 Too Many Requests", "200 OK"],
"archive": ["404 Not Found"],
"old": ["410 Gone"],
"staff": ["403 Forbidden"],
"broken": ["500 Internal Server Error"],
"gateway": ["504 Gateway Timeout", "200 OK"],
"bad-query": ["400 Bad Request"],
}
def flaky(method, path, query, now):
here = now + SKEW["flaky"] # what the flaky host's own clock says
dated = [("Date", http_date(here))]
if path == "/board":
if now < 90: # maintenance: back at 90 s on your clock, which its own clock calls 90 + skew
return answer("503 Service Unavailable", "the price board is being rebuilt\n",
extra=[*dated, ("Retry-After", http_date(90 + SKEW["flaky"]))])
return answer("200 OK", "price board: 40 rows\n", extra=dated)
if path == "/stock":
second = int(now)
PER_SECOND[second] = PER_SECOND.get(second, 0) + 1
if PER_SECOND[second] > 5: # overloaded: no Retry-After, because it cannot know when it will recover
return answer("503 Service Unavailable", "too busy\n", extra=dated)
return answer("200 OK", f"stock of item {query.get('item', '?')}: 12\n", extra=dated)
if path == "/catalogue":
if OUTAGE["on"]:
return answer("503 Service Unavailable", "the catalogue is down\n", extra=dated)
return answer("200 OK", f"catalogue page {query.get('page', '1')}\n", extra=dated)
if path == "/orders" and method == "POST":
return answer("503 Service Unavailable", "orders are paused\n", extra=dated)
name = path.removeprefix("/report/")
if path.startswith("/report/") and name in REPORTS:
n = COUNTS[path] = COUNTS.get(path, 0) + 1
status = REPORTS[name][min(n, len(REPORTS[name])) - 1]
extra = [("Retry-After", "2")] if status.startswith("429") else []
return answer(status, f"{name} report\n" if status.startswith("200") else "", extra=[*dated, *extra])
return answer("404 Not Found", "no such page\n", extra=dated)
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
now = clock.now()
host, method, path = environ.get("HTTP_HOST", ""), environ["REQUEST_METHOD"], environ["PATH_INFO"]
query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
who = environ.get("HTTP_USER_AGENT", "")
if host == SHOP and path == "/api/v1/prices":
status, headers, body = prices(query, who, now)
elif host == FLAKY:
status, headers, body = flaky(method, path, query, now)
elif host == SHOP:
status, headers, body = answer("404 Not Found", "no such page\n")
else:
status, headers, body = answer("421 Misdirected Request", "not a Nimbu Bazaar address\n")
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"t": round(now, 3), "host": host, "method": method, "target": target, "status": int(status[:3]),
"retry_after": dict(headers).get("Retry-After"), "who": who})
start_response(status, headers)
return [body]
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data)}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in headers.items()})
got = {}
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
clock.sleep(LATENCY) # the answer arrives 50 ms after the request left
raw = "".join(f"{k}: {v}\r\n" for k, v in got["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, int(got["status"][:3]))
response.reason = response.msg = got["status"][4:]
return response
http_open = https_open
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network."""
director = urllib.request.OpenerDirector()
for handler in [ShopTransport(), urllib.request.HTTPDefaultErrorHandler(), urllib.request.HTTPRedirectHandler(),
urllib.request.HTTPErrorProcessor(), urllib.request.UnknownHandler(), *handlers]:
director.add_handler(handler)
return director
def get(path, host=SHOP, who="NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)", method="GET"):
"""One request through urllib: (status, headers). urllib raises HTTPError for 4xx and 5xx; this returns them."""
request = urllib.request.Request(f"https://{host}{path}", headers={"User-Agent": who}, method=method,
data=b"{}" if method == "POST" else None)
try:
with OPENER.open(request, timeout=10) as response:
response.read()
return response.status, response.headers
except urllib.error.HTTPError as error:
error.read()
return error.code, error.headers
OPENER = opener()Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Asking again at once finished soonest, in 15.1 s, with 281 refused requests. Waiting for Retry-After cut the
refusals to 14, one for each page that came too early, and took 15.7 s. The two ways that stay under the limit were
never refused and sent exactly 20 requests. That is the aim: a collector paced below the limit never sees a 429, and
its first 429 says its pace is wrong.
A fixed pause is the simplest pace: wait a little longer than the limit allows between requests, here 1.1 s
against the shop’s 1 a second. Add jitter: A random part added to a wait, such as a pause of 1.1 s varied by up to 10 %, so that clients started at the same moment stop sending their requests in step., a random part of the pause, here ±10 %. Many
collectors start at the same second, because a scheduler starts them at the top of a minute; the Amazon Builders’
Library describes clients that line up this way, and adds jitter to timers and periodic jobs to spread them apart.
Scrapy’s DOWNLOAD_DELAY is such a pause between two requests to one domain, and Scrapy 2.19 varies it by ±50 % by
default (DOWNLOAD_DELAY_JITTER).
A token bucket of your own paces the starts of requests instead of the gaps between them. With 3 tokens and 0.9 a second it sent three pages at once, then one every 1.11 s, and finished in 18.9 s against the fixed pause’s 22.4 s. Step through where its waiting happens:
The drawing shows the whole scene; its 6 steps are listed below.
With the drawing focused, the left and right arrow keys step, Home and End go to the first and last step, and Space or K plays and pauses.
Every step as text
- Three at once. Your bucket starts with 3 tokens, so pages 1, 2 and 3 leave at once, 50 ms apart. The API spends a token of its own on each and has about 2 of its 5 left.
- Three answers. All three are answered with 200. Nothing has been refused, and nothing will be: the next decision is taken on your side.
- Wait on your side. Page 4 needs a token your bucket does not have yet. It waits 0.96 s before sending, so the API never sees a request it would have to refuse.
- Page 4 at 1.11 s. Page 4 leaves at 1.11 s. The API has earned a token a second in the meantime and still has 2 left after it.
- One every 1.11 s. From now on a page leaves every 1.11 s, your rate of 0.9 a second. The API earns a token every second, a little faster than you spend them, so its bucket fills up instead of running dry.
- Twenty pages, none refused. Page 20 leaves at 18.89 s. The run sent 20 requests for 20 pages and none was refused; the API still had 3.89 tokens to spare.
Pace at the API's own numbers, then above them
Press Edit on pace.py and set the bucket to rate=1.0, burst=5, the API’s own numbers: in the lab it finishes in
15.1 s with no refusal, because nothing else spends the key. Then set rate=1.2: three requests are refused. A
real key is shared with other jobs and retries, so a real collector keeps a margin, as 0.9 does here.
Checkpoint
Check: A rate limit as a budget of tokens · Pacing below the limit
Retry-After says how long; count a date by the server’s clock
Retry-After comes in two forms, and RFC 9110 defines both: a whole number of seconds, such as 120, or an HTTP date,
the moment when the server expects to be ready again. Sent with 503 Service Unavailable it says how long the service
is expected to be unavailable. A number of seconds is counted from when the answer arrived, so it means the same on
every computer. A date is a moment on the server’s clock, and your clock is not the server’s.
The flaky host closes its price board for 90 s and says until when as a date, by a clock that runs 40 s behind yours.
Two collectors wait for the board: one counts the date from its own clock, the other from the Date header of the
same answer:
limits/retry_after.py
"""Retry-After in its two forms. The board is closed for maintenance for 90 s and says until when as an HTTP date,
by a clock that runs 40 s behind yours. Two clients wait for it: one counts the date from its own clock, the other
from the Date header of the same answer."""
import email.utils
import nimbu_lab as lab
def seconds_to_wait(headers, own_clock):
"""How long Retry-After asks for: a number of seconds, or an HTTP date counted from `own_clock` (epoch seconds)
or, when own_clock is None, from the Date header of the same answer."""
value = headers["Retry-After"].strip()
if value.isdigit():
return int(value)
when = email.utils.parsedate_to_datetime(value).timestamp()
base = own_clock if own_clock is not None else email.utils.parsedate_to_datetime(headers["Date"]).timestamp()
return when - base
def wait_for_board(use_date_header):
lab.reset(skew=-40.0) # the flaky host's clock is 40 s behind yours
waits = []
while True:
status, headers = lab.get("/board", host=lab.FLAKY)
if status == 200:
return len(lab.LOG), waits
own = None if use_date_header else lab.EPOCH + lab.clock.now()
wait = max(1.0, seconds_to_wait(headers, own)) # never ask again sooner than a second
waits.append(round(wait, 2))
lab.clock.sleep(wait)
lab.reset(skew=-40.0)
status, headers = lab.get("/board", host=lab.FLAKY)
print("Your clock:", lab.http_date(0))
print("The answer:", status, "| Date:", headers["Date"], "| Retry-After:", headers["Retry-After"])
print()
for label, use_date in [("counted from your own clock", False), ("counted from its Date header", True)]:
requests, waits = wait_for_board(use_date)
shown = ", ".join(f"{w:g}" for w in waits[:4]) + (", …" if len(waits) > 4 else "")
print(f"{label}: {requests} requests, waits of {shown} s, open at {lab.clock.now():.1f} s")Output
Your clock: Fri, 15 Jan 2027 08:00:00 GMT The answer: 503 | Date: Fri, 15 Jan 2027 07:59:20 GMT | Retry-After: Fri, 15 Jan 2027 08:00:50 GMT counted from your own clock: 41 requests, waits of 49.95, 1, 1, 1, … s, open at 91.0 s counted from its Date header: 2 requests, waits of 90 s, open at 90.1 s
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 retry_after.py
nimbu_lab.py (the shop) · limits/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, with the pages of the rate-limit lesson. Every product, price and
person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers, and one change: the last step hands each request to app() instead of opening
a socket, so nothing leaves your program. Time is the shop's clock. Each request takes 50 ms of it, and sleep() moves
it on instead of waiting, so a run that spans minutes ends at once and prints the same numbers every time.
Two hosts answer:
bazaar.nimbu.test the price API, /api/v1/prices?page=1 to 20: a token bucket per client, 5 requests at once and
1 more each second, then 429 Too Many Requests with Retry-After
flaky.nimbu.test /board (closed for maintenance for the first 90 s, 503 with Retry-After as an HTTP date),
/stock?item=n (serves 5 requests in each second and answers the rest 503), /report/<name>
(each name fails in its own way, see REPORTS) and /catalogue?page=n (503 to everything while
OUTAGE["on"] is True)
"""
import email.utils
import http.client
import io
import json
import math
import urllib.error
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
SHOP = "bazaar.nimbu.test"
FLAKY = "flaky.nimbu.test"
EPOCH = 1_800_000_000 # the clock reads 0 at this moment, in seconds since 1970 (what Date headers are made from)
LATENCY = 0.05 # seconds a request takes, there and back
class Clock:
"""The shop's clock, in seconds. sleep() moves it on instead of waiting."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += max(0.0, seconds)
clock = Clock()
LOG = [] # what the shop saw: one entry per request
SKEW = {"flaky": 0.0} # how far the flaky host's clock is from yours: -40.0 means 40 s behind
COUNTS = {} # attempts per /report/ path, so a page can fail twice and then work
OUTAGE = {"on": False} # True: the flaky host's catalogue is down
class Bucket:
"""The API's limit for one client: at most `capacity` tokens, `rate` new tokens a second, one per request."""
def __init__(self, capacity, rate, now):
self.capacity, self.rate, self.tokens, self.at = capacity, rate, float(capacity), now
def take(self, now):
"""(True, 0) when a token was taken; (False, seconds until the next one) when the bucket is empty."""
self.tokens = min(self.capacity, self.tokens + (now - self.at) * self.rate)
self.at = now
if self.tokens >= 1 - 1e-9:
self.tokens -= 1
return True, 0.0
return False, (1 - self.tokens) / self.rate
BUCKETS = {} # one Bucket per client, told apart by User-Agent
PER_SECOND = {} # /stock requests seen in each whole second
def reset(skew=0.0):
"""A fresh shop: clock at 0, empty log, full buckets; the flaky host's clock `skew` seconds from yours."""
clock.seconds = 0.0
for store in (LOG, BUCKETS, COUNTS, PER_SECOND):
store.clear()
SKEW["flaky"] = skew
OUTAGE["on"] = False
def http_date(seconds):
"""An HTTP date for a moment on the shop's clock, as Date and Retry-After carry it."""
return email.utils.formatdate(EPOCH + seconds, usegmt=True)
def answer(status, body="", kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def prices(query, who, now):
page = int(query.get("page", "1"))
if not 1 <= page <= 20:
return answer("404 Not Found", "no such page\n")
bucket = BUCKETS.setdefault(who, Bucket(5, 1.0, now))
ok, wait = bucket.take(now)
if not ok: # RFC 6585: 429 may say how long to wait; delay-seconds is a whole number
return answer("429 Too Many Requests", json.dumps({"error": "5 requests at once, then 1 a second"}),
"application/json", [("Retry-After", str(math.ceil(wait)))])
rows = [{"item": f"NB-{(page - 1) * 3 + i + 1:04d}", "paise": 2500 + 75 * ((page * 3 + i) % 40)} for i in range(3)]
return answer("200 OK", json.dumps({"page": page, "prices": rows}), "application/json")
REPORTS = { # /report/<name>: the answers each one gives, attempt after attempt (the last one repeats)
"daily": ["200 OK"],
"weekly": ["503 Service Unavailable", "503 Service Unavailable", "200 OK"],
"monthly": ["429 Too Many Requests", "200 OK"],
"archive": ["404 Not Found"],
"old": ["410 Gone"],
"staff": ["403 Forbidden"],
"broken": ["500 Internal Server Error"],
"gateway": ["504 Gateway Timeout", "200 OK"],
"bad-query": ["400 Bad Request"],
}
def flaky(method, path, query, now):
here = now + SKEW["flaky"] # what the flaky host's own clock says
dated = [("Date", http_date(here))]
if path == "/board":
if now < 90: # maintenance: back at 90 s on your clock, which its own clock calls 90 + skew
return answer("503 Service Unavailable", "the price board is being rebuilt\n",
extra=[*dated, ("Retry-After", http_date(90 + SKEW["flaky"]))])
return answer("200 OK", "price board: 40 rows\n", extra=dated)
if path == "/stock":
second = int(now)
PER_SECOND[second] = PER_SECOND.get(second, 0) + 1
if PER_SECOND[second] > 5: # overloaded: no Retry-After, because it cannot know when it will recover
return answer("503 Service Unavailable", "too busy\n", extra=dated)
return answer("200 OK", f"stock of item {query.get('item', '?')}: 12\n", extra=dated)
if path == "/catalogue":
if OUTAGE["on"]:
return answer("503 Service Unavailable", "the catalogue is down\n", extra=dated)
return answer("200 OK", f"catalogue page {query.get('page', '1')}\n", extra=dated)
if path == "/orders" and method == "POST":
return answer("503 Service Unavailable", "orders are paused\n", extra=dated)
name = path.removeprefix("/report/")
if path.startswith("/report/") and name in REPORTS:
n = COUNTS[path] = COUNTS.get(path, 0) + 1
status = REPORTS[name][min(n, len(REPORTS[name])) - 1]
extra = [("Retry-After", "2")] if status.startswith("429") else []
return answer(status, f"{name} report\n" if status.startswith("200") else "", extra=[*dated, *extra])
return answer("404 Not Found", "no such page\n", extra=dated)
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
now = clock.now()
host, method, path = environ.get("HTTP_HOST", ""), environ["REQUEST_METHOD"], environ["PATH_INFO"]
query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
who = environ.get("HTTP_USER_AGENT", "")
if host == SHOP and path == "/api/v1/prices":
status, headers, body = prices(query, who, now)
elif host == FLAKY:
status, headers, body = flaky(method, path, query, now)
elif host == SHOP:
status, headers, body = answer("404 Not Found", "no such page\n")
else:
status, headers, body = answer("421 Misdirected Request", "not a Nimbu Bazaar address\n")
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"t": round(now, 3), "host": host, "method": method, "target": target, "status": int(status[:3]),
"retry_after": dict(headers).get("Retry-After"), "who": who})
start_response(status, headers)
return [body]
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data)}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in headers.items()})
got = {}
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
clock.sleep(LATENCY) # the answer arrives 50 ms after the request left
raw = "".join(f"{k}: {v}\r\n" for k, v in got["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, int(got["status"][:3]))
response.reason = response.msg = got["status"][4:]
return response
http_open = https_open
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network."""
director = urllib.request.OpenerDirector()
for handler in [ShopTransport(), urllib.request.HTTPDefaultErrorHandler(), urllib.request.HTTPRedirectHandler(),
urllib.request.HTTPErrorProcessor(), urllib.request.UnknownHandler(), *handlers]:
director.add_handler(handler)
return director
def get(path, host=SHOP, who="NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)", method="GET"):
"""One request through urllib: (status, headers). urllib raises HTTPError for 4xx and 5xx; this returns them."""
request = urllib.request.Request(f"https://{host}{path}", headers={"User-Agent": who}, method=method,
data=b"{}" if method == "POST" else None)
try:
with OPENER.open(request, timeout=10) as response:
response.read()
return response.status, response.headers
except urllib.error.HTTPError as error:
error.read()
return error.code, error.headers
OPENER = opener()Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Counted from your own clock, the date was 50 s away, so the first collector came back 40 s early; after that the
date looked past, and with a floor of one second it asked again and again: 41 requests. The second subtracted the
answer’s Date from its Retry-After, both read from the same clock, got exactly 90 s and needed 2 requests. Step
through both:
The drawing shows the whole scene; its 6 steps are listed below.
With the drawing focused, the left and right arrow keys step, Home and End go to the first and last step, and Space or K plays and pauses.
Every step as text
- The board is closed. You ask for /board at 08:00:00 by your clock. The host is rebuilding it for 90 s, until 08:01:30 by your clock.
- 503 with a date. The host answers 503 with Date 07:59:20 and Retry-After 08:00:50, both read from its own clock, which runs 40 s behind yours.
- Counted from your clock. 08:00:50 minus your 08:00:00 is 50 s. The collector comes back after 50 s, when the host is 50 s into its 90 s of work.
- The date looks past. The second 503 names the same 08:00:50, which your clock has already passed, so the wait comes out negative. With a floor of 1 s, the collector asks every second until 90 s: 41 requests in all.
- Counted from its Date. A second collector subtracts the answer's own Date instead: 08:00:50 minus 07:59:20 is 90 s, the length of the maintenance, whatever either clock says.
- Open at 90 s. It asks once more at 90 s and gets the board: 2 requests in all, the second one at the right moment.
The idea is not new: RFC 9111 computes how long a cached response stays fresh as Expires minus Date for the same
reason, “to reduce clock skew” by using the origin server’s own clock. One caution: RFC 9110 lets a server leave out
Date on 5xx answers, so fall back to your own clock when it is missing.
A long Retry-After is a reason to move on
urllib3’s Retry obeys Retry-After on 413, 429 and 503 answers by default and caps it at 21,600 s, six hours.
In a crawl of many hosts, do not sleep through a wait of an hour: park that host’s queue until the time it named,
as the breaker of the blocks lesson parks a host, and keep working on the others.
Without Retry-After, back off exponentially and add jitter
An overloaded host often cannot say when it will recover, so it answers 503 with no Retry-After. Fifty collectors
start in the same second against such a host, which serves 5 requests a second and refuses the rest, and each retries
until it has its page:
limits/backoff.py
"""Fifty collectors start in the same second against a host that serves 5 requests a second and answers the rest
503 without Retry-After. Each retries until it gets its page: after a fixed second, with exponential backoff from
2 s, or with exponential backoff and full jitter. The shop's log says what each way cost the host."""
import heapq
import random
import nimbu_lab as lab
CLIENTS, BASE, CAP = 50, 2.0, 30.0
def fixed(failures, rng):
return 1.0
def exponential(failures, rng):
return min(CAP, BASE * 2 ** (failures - 1)) # 2, 4, 8, 16, 30, 30 … seconds
def full_jitter(failures, rng):
return rng.uniform(0, min(CAP, BASE * 2 ** (failures - 1))) # anywhere from 0 to that window
def run(delay):
lab.reset()
rng = random.Random(42) # seeded, so every run draws the same waits
queue = [(0.0, client, 0) for client in range(CLIENTS)] # (when it asks, which collector, failures so far)
heapq.heapify(queue)
done_at = 0.0
while queue:
when, client, failures = heapq.heappop(queue)
lab.clock.seconds = when # the collectors share the shop's clock
status, _ = lab.get(f"/stock?item={client}", host=lab.FLAKY, who=f"collector-{client}")
if status == 200:
done_at = max(done_at, when + lab.LATENCY)
else:
heapq.heappush(queue, (when + lab.LATENCY + delay(failures + 1, rng), client, failures + 1))
per_second = {}
for entry in lab.LOG:
per_second[int(entry["t"])] = per_second.get(int(entry["t"]), 0) + 1
return len(lab.LOG), done_at, per_second
results = {name: run(way) for name, way in [("fixed 1 s", fixed), ("exponential", exponential),
("full jitter", full_jitter)]}
print(f"{'retry wait':<13}{'requests':>9}{'refused':>9}{'all served at':>15}")
for name, (requests, done_at, _) in results.items():
print(f"{name:<13}{requests:>9}{requests - CLIENTS:>9}{done_at:>14.1f}s")
print()
print("Requests that reached the host in each of the first 16 seconds (it serves 5):")
print(f"{'second':<12}", " ".join(f"{s:>2}" for s in range(16)))
for name, (_, _, per_second) in results.items():
print(f"{name:<12}", " ".join(f"{per_second.get(s, 0):>2}" for s in range(16)))Output
retry wait requests refused all served at fixed 1 s 275 225 9.5s exponential 275 225 180.5s full jitter 168 118 19.5s Requests that reached the host in each of the first 16 seconds (it serves 5): second 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 fixed 1 s 50 45 40 35 30 25 20 15 10 5 0 0 0 0 0 0 exponential 50 0 45 0 0 0 40 0 0 0 0 0 0 0 35 0 full jitter 72 34 15 14 5 7 1 5 6 1 2 2 0 0 1 0
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 backoff.py
nimbu_lab.py (the shop) · limits/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, with the pages of the rate-limit lesson. Every product, price and
person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers, and one change: the last step hands each request to app() instead of opening
a socket, so nothing leaves your program. Time is the shop's clock. Each request takes 50 ms of it, and sleep() moves
it on instead of waiting, so a run that spans minutes ends at once and prints the same numbers every time.
Two hosts answer:
bazaar.nimbu.test the price API, /api/v1/prices?page=1 to 20: a token bucket per client, 5 requests at once and
1 more each second, then 429 Too Many Requests with Retry-After
flaky.nimbu.test /board (closed for maintenance for the first 90 s, 503 with Retry-After as an HTTP date),
/stock?item=n (serves 5 requests in each second and answers the rest 503), /report/<name>
(each name fails in its own way, see REPORTS) and /catalogue?page=n (503 to everything while
OUTAGE["on"] is True)
"""
import email.utils
import http.client
import io
import json
import math
import urllib.error
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
SHOP = "bazaar.nimbu.test"
FLAKY = "flaky.nimbu.test"
EPOCH = 1_800_000_000 # the clock reads 0 at this moment, in seconds since 1970 (what Date headers are made from)
LATENCY = 0.05 # seconds a request takes, there and back
class Clock:
"""The shop's clock, in seconds. sleep() moves it on instead of waiting."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += max(0.0, seconds)
clock = Clock()
LOG = [] # what the shop saw: one entry per request
SKEW = {"flaky": 0.0} # how far the flaky host's clock is from yours: -40.0 means 40 s behind
COUNTS = {} # attempts per /report/ path, so a page can fail twice and then work
OUTAGE = {"on": False} # True: the flaky host's catalogue is down
class Bucket:
"""The API's limit for one client: at most `capacity` tokens, `rate` new tokens a second, one per request."""
def __init__(self, capacity, rate, now):
self.capacity, self.rate, self.tokens, self.at = capacity, rate, float(capacity), now
def take(self, now):
"""(True, 0) when a token was taken; (False, seconds until the next one) when the bucket is empty."""
self.tokens = min(self.capacity, self.tokens + (now - self.at) * self.rate)
self.at = now
if self.tokens >= 1 - 1e-9:
self.tokens -= 1
return True, 0.0
return False, (1 - self.tokens) / self.rate
BUCKETS = {} # one Bucket per client, told apart by User-Agent
PER_SECOND = {} # /stock requests seen in each whole second
def reset(skew=0.0):
"""A fresh shop: clock at 0, empty log, full buckets; the flaky host's clock `skew` seconds from yours."""
clock.seconds = 0.0
for store in (LOG, BUCKETS, COUNTS, PER_SECOND):
store.clear()
SKEW["flaky"] = skew
OUTAGE["on"] = False
def http_date(seconds):
"""An HTTP date for a moment on the shop's clock, as Date and Retry-After carry it."""
return email.utils.formatdate(EPOCH + seconds, usegmt=True)
def answer(status, body="", kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def prices(query, who, now):
page = int(query.get("page", "1"))
if not 1 <= page <= 20:
return answer("404 Not Found", "no such page\n")
bucket = BUCKETS.setdefault(who, Bucket(5, 1.0, now))
ok, wait = bucket.take(now)
if not ok: # RFC 6585: 429 may say how long to wait; delay-seconds is a whole number
return answer("429 Too Many Requests", json.dumps({"error": "5 requests at once, then 1 a second"}),
"application/json", [("Retry-After", str(math.ceil(wait)))])
rows = [{"item": f"NB-{(page - 1) * 3 + i + 1:04d}", "paise": 2500 + 75 * ((page * 3 + i) % 40)} for i in range(3)]
return answer("200 OK", json.dumps({"page": page, "prices": rows}), "application/json")
REPORTS = { # /report/<name>: the answers each one gives, attempt after attempt (the last one repeats)
"daily": ["200 OK"],
"weekly": ["503 Service Unavailable", "503 Service Unavailable", "200 OK"],
"monthly": ["429 Too Many Requests", "200 OK"],
"archive": ["404 Not Found"],
"old": ["410 Gone"],
"staff": ["403 Forbidden"],
"broken": ["500 Internal Server Error"],
"gateway": ["504 Gateway Timeout", "200 OK"],
"bad-query": ["400 Bad Request"],
}
def flaky(method, path, query, now):
here = now + SKEW["flaky"] # what the flaky host's own clock says
dated = [("Date", http_date(here))]
if path == "/board":
if now < 90: # maintenance: back at 90 s on your clock, which its own clock calls 90 + skew
return answer("503 Service Unavailable", "the price board is being rebuilt\n",
extra=[*dated, ("Retry-After", http_date(90 + SKEW["flaky"]))])
return answer("200 OK", "price board: 40 rows\n", extra=dated)
if path == "/stock":
second = int(now)
PER_SECOND[second] = PER_SECOND.get(second, 0) + 1
if PER_SECOND[second] > 5: # overloaded: no Retry-After, because it cannot know when it will recover
return answer("503 Service Unavailable", "too busy\n", extra=dated)
return answer("200 OK", f"stock of item {query.get('item', '?')}: 12\n", extra=dated)
if path == "/catalogue":
if OUTAGE["on"]:
return answer("503 Service Unavailable", "the catalogue is down\n", extra=dated)
return answer("200 OK", f"catalogue page {query.get('page', '1')}\n", extra=dated)
if path == "/orders" and method == "POST":
return answer("503 Service Unavailable", "orders are paused\n", extra=dated)
name = path.removeprefix("/report/")
if path.startswith("/report/") and name in REPORTS:
n = COUNTS[path] = COUNTS.get(path, 0) + 1
status = REPORTS[name][min(n, len(REPORTS[name])) - 1]
extra = [("Retry-After", "2")] if status.startswith("429") else []
return answer(status, f"{name} report\n" if status.startswith("200") else "", extra=[*dated, *extra])
return answer("404 Not Found", "no such page\n", extra=dated)
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
now = clock.now()
host, method, path = environ.get("HTTP_HOST", ""), environ["REQUEST_METHOD"], environ["PATH_INFO"]
query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
who = environ.get("HTTP_USER_AGENT", "")
if host == SHOP and path == "/api/v1/prices":
status, headers, body = prices(query, who, now)
elif host == FLAKY:
status, headers, body = flaky(method, path, query, now)
elif host == SHOP:
status, headers, body = answer("404 Not Found", "no such page\n")
else:
status, headers, body = answer("421 Misdirected Request", "not a Nimbu Bazaar address\n")
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"t": round(now, 3), "host": host, "method": method, "target": target, "status": int(status[:3]),
"retry_after": dict(headers).get("Retry-After"), "who": who})
start_response(status, headers)
return [body]
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data)}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in headers.items()})
got = {}
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
clock.sleep(LATENCY) # the answer arrives 50 ms after the request left
raw = "".join(f"{k}: {v}\r\n" for k, v in got["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, int(got["status"][:3]))
response.reason = response.msg = got["status"][4:]
return response
http_open = https_open
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network."""
director = urllib.request.OpenerDirector()
for handler in [ShopTransport(), urllib.request.HTTPDefaultErrorHandler(), urllib.request.HTTPRedirectHandler(),
urllib.request.HTTPErrorProcessor(), urllib.request.UnknownHandler(), *handlers]:
director.add_handler(handler)
return director
def get(path, host=SHOP, who="NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)", method="GET"):
"""One request through urllib: (status, headers). urllib raises HTTPError for 4xx and 5xx; this returns them."""
request = urllib.request.Request(f"https://{host}{path}", headers={"User-Agent": who}, method=method,
data=b"{}" if method == "POST" else None)
try:
with OPENER.open(request, timeout=10) as response:
response.read()
return response.status, response.headers
except urllib.error.HTTPError as error:
error.read()
return error.code, error.headers
OPENER = opener()Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Retrying every second finished first, at 9.5 s, by keeping the host full: it refused 225 requests while it was already overloaded. exponential backoff: Waiting longer after each failure before trying again, the wait doubling each time up to a cap, such as 2, 4, 8 and 16 s up to 30 s. doubles the wait after every failure, 2, 4, 8 and 16 s up to a cap of 30 s, yet it sent just as many requests and took 180.5 s. The table of seconds shows why: collectors that failed together waited the same time and came back together, in waves of 45, 40 and 35 at seconds 2, 6 and 14.
Full jitter breaks the waves. Each wait is drawn at random between 0 and the doubling window, so the first retry falls anywhere in the next 2 s, the second anywhere in the next 4 s:
Here n is the number of failures so far, base 2 s and cap 30 s: after the third failure the wait is somewhere
between 0 and 8 s. With it the host received 168 requests instead of 275, and every collector had its page by 19.5 s.
This is the algorithm the AWS Architecture Blog named Full Jitter, and Google’s SRE book gives the rule plainly:
always use randomized exponential backoff when scheduling retries.
Libraries offer both kinds, so check which one you configured. Tenacity’s wait_random_exponential is Full Jitter,
and its documentation says that wait_exponential, which has no jitter, is not suitable for contention between
processes for a shared resource. urllib3’s Retry waits backoff_factor × 2 ** (retries so far), capped by
backoff_max (120 s by default), and adds jitter only if you set backoff_jitter.
A retry decorator with no stop and no wait
Tenacity’s documentation says it: its default behaviour is to retry forever without waiting. @retry on its own
turns one failure into an endless stream of requests. Give every retry a wait and a stop, for example
wait_random_exponential(multiplier=2, max=30) and stop_after_attempt(3).
Checkpoint
Check: Retry-After in seconds or as a date · Exponential backoff with full jitter
Retry only what can succeed, and give the whole run a retry budget
Some failures change if you ask again later and some never will. Ten requests to the flaky host, each failing in its own way, go through one policy: retry only temporary answers, only for requests that are safe to send twice, at most three tries each:
limits/retry_policy.py
"""What to retry, how often, and how many retries a whole run may spend. First, nine reports of the flaky host and
one POST, each failing in its own way, through one policy (the refusal comes last, because it stops the host).
Then an outage: 100 catalogue pages while the host answers 503 to everything, with retries capped per request,
then also by a budget for the whole run."""
import random
import nimbu_lab as lab
RETRY = {408, 429, 500, 502, 503, 504} # may work if asked again later
STOP = {401, 403} # the site refused: stop asking this host (the blocks lesson's breaker)
SAFE = {"GET", "HEAD"} # methods a client may repeat by itself (RFC 9110, section 9.2.2)
class Policy:
"""At most `attempts` tries a request; with `budget`, retries at most that share of the run's first tries."""
def __init__(self, attempts=3, budget=None):
self.attempts, self.budget, self.rng = attempts, budget, random.Random(3)
self.first = self.retries = 0
def wait(self, headers, attempt):
value = (headers.get("Retry-After") or "").strip()
if value.isdigit():
return int(value)
return self.rng.uniform(0, min(30.0, 2 ** attempt)) # full jitter: 0-2 s, 0-4 s, 0-8 s …
def may_retry(self, method, status, attempt):
if method not in SAFE or status not in RETRY or attempt >= self.attempts:
return False
return self.budget is None or self.retries + 1 <= self.budget * self.first
def fetch(path, policy, method="GET"):
policy.first += 1
seen = []
for attempt in range(1, policy.attempts + 1):
status, headers = lab.get(path, host=lab.FLAKY, method=method)
seen.append(status)
if 200 <= status < 300:
return seen, "saved"
if status in STOP:
return seen, "stop the host"
if status in (404, 410):
return seen, "skip: nothing there"
if status not in RETRY:
return seen, "skip: fix the request"
if not policy.may_retry(method, status, attempt):
return seen, f"give up for now{' (a POST is never retried)' if method not in SAFE else ''}"
policy.retries += 1
lab.clock.sleep(policy.wait(headers, attempt))
return seen, "give up for now"
lab.reset()
policy = Policy()
print(f"{'request':<23}{'answers':<14}what the collector does")
for method, path in [*(("GET", f"/report/{name}") for name in lab.REPORTS if name != "staff"),
("POST", "/orders"), ("GET", "/report/staff")]:
seen, outcome = fetch(path, policy, method)
print(f"{method + ' ' + path:<23}{' '.join(map(str, seen)):<14}{outcome}")
print(f"{policy.first} first tries and {policy.retries} retries: {len(lab.LOG)} requests in all")
print()
for label, budget in [("3 tries a page", None), ("3 tries a page, retries within 10 %", 0.10)]:
lab.reset()
lab.OUTAGE["on"] = True
policy = Policy(budget=budget)
outcomes = [fetch(f"/catalogue?page={page}", policy)[1] for page in range(1, 101)]
print(f"Outage, {label}: the host got {len(lab.LOG)} requests ({policy.retries} retries) "
f"for 100 pages; {outcomes.count('give up for now')} left for a later run")Output
request answers what the collector does GET /report/daily 200 saved GET /report/weekly 503 503 200 saved GET /report/monthly 429 200 saved GET /report/archive 404 skip: nothing there GET /report/old 410 skip: nothing there GET /report/broken 500 500 500 give up for now GET /report/gateway 504 200 saved GET /report/bad-query 400 skip: fix the request POST /orders 503 give up for now (a POST is never retried) GET /report/staff 403 stop the host 10 first tries and 6 retries: 16 requests in all Outage, 3 tries a page: the host got 300 requests (200 retries) for 100 pages; 100 left for a later run Outage, 3 tries a page, retries within 10 %: the host got 110 requests (10 retries) for 100 pages; 100 left for a later run
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 retry_policy.py
nimbu_lab.py (the shop) · limits/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, with the pages of the rate-limit lesson. Every product, price and
person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers, and one change: the last step hands each request to app() instead of opening
a socket, so nothing leaves your program. Time is the shop's clock. Each request takes 50 ms of it, and sleep() moves
it on instead of waiting, so a run that spans minutes ends at once and prints the same numbers every time.
Two hosts answer:
bazaar.nimbu.test the price API, /api/v1/prices?page=1 to 20: a token bucket per client, 5 requests at once and
1 more each second, then 429 Too Many Requests with Retry-After
flaky.nimbu.test /board (closed for maintenance for the first 90 s, 503 with Retry-After as an HTTP date),
/stock?item=n (serves 5 requests in each second and answers the rest 503), /report/<name>
(each name fails in its own way, see REPORTS) and /catalogue?page=n (503 to everything while
OUTAGE["on"] is True)
"""
import email.utils
import http.client
import io
import json
import math
import urllib.error
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
SHOP = "bazaar.nimbu.test"
FLAKY = "flaky.nimbu.test"
EPOCH = 1_800_000_000 # the clock reads 0 at this moment, in seconds since 1970 (what Date headers are made from)
LATENCY = 0.05 # seconds a request takes, there and back
class Clock:
"""The shop's clock, in seconds. sleep() moves it on instead of waiting."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += max(0.0, seconds)
clock = Clock()
LOG = [] # what the shop saw: one entry per request
SKEW = {"flaky": 0.0} # how far the flaky host's clock is from yours: -40.0 means 40 s behind
COUNTS = {} # attempts per /report/ path, so a page can fail twice and then work
OUTAGE = {"on": False} # True: the flaky host's catalogue is down
class Bucket:
"""The API's limit for one client: at most `capacity` tokens, `rate` new tokens a second, one per request."""
def __init__(self, capacity, rate, now):
self.capacity, self.rate, self.tokens, self.at = capacity, rate, float(capacity), now
def take(self, now):
"""(True, 0) when a token was taken; (False, seconds until the next one) when the bucket is empty."""
self.tokens = min(self.capacity, self.tokens + (now - self.at) * self.rate)
self.at = now
if self.tokens >= 1 - 1e-9:
self.tokens -= 1
return True, 0.0
return False, (1 - self.tokens) / self.rate
BUCKETS = {} # one Bucket per client, told apart by User-Agent
PER_SECOND = {} # /stock requests seen in each whole second
def reset(skew=0.0):
"""A fresh shop: clock at 0, empty log, full buckets; the flaky host's clock `skew` seconds from yours."""
clock.seconds = 0.0
for store in (LOG, BUCKETS, COUNTS, PER_SECOND):
store.clear()
SKEW["flaky"] = skew
OUTAGE["on"] = False
def http_date(seconds):
"""An HTTP date for a moment on the shop's clock, as Date and Retry-After carry it."""
return email.utils.formatdate(EPOCH + seconds, usegmt=True)
def answer(status, body="", kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def prices(query, who, now):
page = int(query.get("page", "1"))
if not 1 <= page <= 20:
return answer("404 Not Found", "no such page\n")
bucket = BUCKETS.setdefault(who, Bucket(5, 1.0, now))
ok, wait = bucket.take(now)
if not ok: # RFC 6585: 429 may say how long to wait; delay-seconds is a whole number
return answer("429 Too Many Requests", json.dumps({"error": "5 requests at once, then 1 a second"}),
"application/json", [("Retry-After", str(math.ceil(wait)))])
rows = [{"item": f"NB-{(page - 1) * 3 + i + 1:04d}", "paise": 2500 + 75 * ((page * 3 + i) % 40)} for i in range(3)]
return answer("200 OK", json.dumps({"page": page, "prices": rows}), "application/json")
REPORTS = { # /report/<name>: the answers each one gives, attempt after attempt (the last one repeats)
"daily": ["200 OK"],
"weekly": ["503 Service Unavailable", "503 Service Unavailable", "200 OK"],
"monthly": ["429 Too Many Requests", "200 OK"],
"archive": ["404 Not Found"],
"old": ["410 Gone"],
"staff": ["403 Forbidden"],
"broken": ["500 Internal Server Error"],
"gateway": ["504 Gateway Timeout", "200 OK"],
"bad-query": ["400 Bad Request"],
}
def flaky(method, path, query, now):
here = now + SKEW["flaky"] # what the flaky host's own clock says
dated = [("Date", http_date(here))]
if path == "/board":
if now < 90: # maintenance: back at 90 s on your clock, which its own clock calls 90 + skew
return answer("503 Service Unavailable", "the price board is being rebuilt\n",
extra=[*dated, ("Retry-After", http_date(90 + SKEW["flaky"]))])
return answer("200 OK", "price board: 40 rows\n", extra=dated)
if path == "/stock":
second = int(now)
PER_SECOND[second] = PER_SECOND.get(second, 0) + 1
if PER_SECOND[second] > 5: # overloaded: no Retry-After, because it cannot know when it will recover
return answer("503 Service Unavailable", "too busy\n", extra=dated)
return answer("200 OK", f"stock of item {query.get('item', '?')}: 12\n", extra=dated)
if path == "/catalogue":
if OUTAGE["on"]:
return answer("503 Service Unavailable", "the catalogue is down\n", extra=dated)
return answer("200 OK", f"catalogue page {query.get('page', '1')}\n", extra=dated)
if path == "/orders" and method == "POST":
return answer("503 Service Unavailable", "orders are paused\n", extra=dated)
name = path.removeprefix("/report/")
if path.startswith("/report/") and name in REPORTS:
n = COUNTS[path] = COUNTS.get(path, 0) + 1
status = REPORTS[name][min(n, len(REPORTS[name])) - 1]
extra = [("Retry-After", "2")] if status.startswith("429") else []
return answer(status, f"{name} report\n" if status.startswith("200") else "", extra=[*dated, *extra])
return answer("404 Not Found", "no such page\n", extra=dated)
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
now = clock.now()
host, method, path = environ.get("HTTP_HOST", ""), environ["REQUEST_METHOD"], environ["PATH_INFO"]
query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
who = environ.get("HTTP_USER_AGENT", "")
if host == SHOP and path == "/api/v1/prices":
status, headers, body = prices(query, who, now)
elif host == FLAKY:
status, headers, body = flaky(method, path, query, now)
elif host == SHOP:
status, headers, body = answer("404 Not Found", "no such page\n")
else:
status, headers, body = answer("421 Misdirected Request", "not a Nimbu Bazaar address\n")
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"t": round(now, 3), "host": host, "method": method, "target": target, "status": int(status[:3]),
"retry_after": dict(headers).get("Retry-After"), "who": who})
start_response(status, headers)
return [body]
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data)}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in headers.items()})
got = {}
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
clock.sleep(LATENCY) # the answer arrives 50 ms after the request left
raw = "".join(f"{k}: {v}\r\n" for k, v in got["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BytesIO(body), message, req.full_url, int(got["status"][:3]))
response.reason = response.msg = got["status"][4:]
return response
http_open = https_open
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network."""
director = urllib.request.OpenerDirector()
for handler in [ShopTransport(), urllib.request.HTTPDefaultErrorHandler(), urllib.request.HTTPRedirectHandler(),
urllib.request.HTTPErrorProcessor(), urllib.request.UnknownHandler(), *handlers]:
director.add_handler(handler)
return director
def get(path, host=SHOP, who="NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)", method="GET"):
"""One request through urllib: (status, headers). urllib raises HTTPError for 4xx and 5xx; this returns them."""
request = urllib.request.Request(f"https://{host}{path}", headers={"User-Agent": who}, method=method,
data=b"{}" if method == "POST" else None)
try:
with OPENER.open(request, timeout=10) as response:
response.read()
return response.status, response.headers
except urllib.error.HTTPError as error:
error.read()
return error.code, error.headers
OPENER = opener()Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The temporary answers are 408, 429, 500, 502, 503 and 504: a busy, broken or unreachable server may answer the same
request later, and RFC 9110 says a client that got 408 may repeat its request. Scrapy retries the same codes, plus two
that some proxies send, RETRY_TIMES (2) times after the first download. The other answers are final. 404 and 410 say
there is nothing at the address, 400 says the request itself is wrong, and 401 and 403 are a refusal that stops the
host. A request that is not safe to repeat is not retried by the client at all: RFC 9110 says a client should not
automatically retry a non-idempotent method such as POST, because it may place a second order. GET and HEAD are
idempotent method: A request method whose intended effect is the same whether a request is sent once or several times, such as GET, HEAD, PUT and DELETE in RFC 9110, so a client may repeat it after a failure. POST is not one.. Follow four of the requests through the policy:
The drawing shows the whole scene; its 7 steps are listed below.
With the drawing focused, the left and right arrow keys step, Home and End go to the first and last step, and Space or K plays and pauses.
Every step as text
- weekly answers 503. GET /report/weekly answers 503. That is not a final answer: a busy server may answer the same request later.
- Temporary, and safe. 503 is one of the answers that can change, and a GET may be sent twice without doing anything twice, so the policy looks at what it has left.
- A try to spare. This was try 1 of 3 and the run's budget has retries left. No Retry-After came, so the policy draws a wait between 0 and 2 s.
- Saved on the third try. Try 2 gets 503 again and waits between 0 and 4 s; try 3 gets 200, a final answer, and the page is saved.
- A POST is not repeated. POST /orders also answers 503, but a POST may place an order each time it is sent, so the client does not repeat it by itself: it is kept for later.
- A refusal stops the host. GET /report/staff answers 403, a refusal. That is a final answer, and the breaker of the blocks lesson stops every request to this host.
- The budget runs out. In the outage every page fails. Once retries reach 10 % of the run's first tries, a failing page gets no retry and waits for the next run: 110 requests reach the host instead of 300.
Limits per request are not enough when a whole host fails. In the outage every page failed, and three tries a page sent 300 requests for 100 pages, tripling the load on a host that was already down. Google’s SRE book describes a per-request limit of three tries together with a retry budget: A limit on retries for a whole run or client rather than per request, such as retries no more than 10 % of first tries, so that a failing host gets little extra load. per client: retry only while retries stay below 10 % of requests, which it reports cut the growth from nearly 3 times to 1.1 times. The same budget sent 110 requests. Retries also multiply across layers: if three layers each try 4 times, one action can reach the bottom layer 64 times, so retry in one place only.
Not affiliated
Based on public information about GitHub. MySmartCoPilot is not affiliated with GitHub, and GitHub’s real REST API limits may differ from what this lesson describes.
Read the answer before you classify it. GitHub’s REST API, for example, reports its secondary rate limits as 403
or 429 with a message, and asks for the full retry-after, or at least a minute without one, before trying again.
Log every retry with its address, status, try number and wait: a rising count of retries is the first sign that a
pace or a policy is wrong.
Key idea
Retry only answers that can change, only for requests that are safe to repeat, a few times each and within a budget for the whole run, and wait between tries as long as Retry-After asks or, without it, a jittered and growing time.
Checkpoint
Check: What to retry, and a budget for retries
Practise: build a client that waits its turn
The lab builds the client of this lesson in two steps. First the decision after a failure: how long to wait, from
either form of Retry-After or from full jitter, or not to retry at all. Then the client around it, which paces
every request with its own bucket and stops after a set number of tries. The sample tests read the moment each
request reached a host whose clock never waits:
Code lab · 2 steps
A polite client in two steps
Write the wait that follows a failure, then a client that paces every request and retries within limits.
- Decide how long to wait before a retry, or not to retry To do
- A client that paces itself and retries within limits To do
Exercise · Medium · Python
Decide how long to wait before a retry, or not to retry
Write retry_delay(status, headers, attempt, now, rng, base=2.0, cap=30.0, tries=3) in retrydelay.py. It gets the answer to a GET request that just failed and returns how many seconds to wait before the next try, or None when the request should not be tried again now:
statusis the status code andheadersa dictionary of the answer's headers, whose names may come in any case;attemptis the try that just failed,1for the first;triesis how many tries a request gets in all;nowis your own clock in epoch seconds, andrngarandom.Randomto draw the jitter from.
The rules, in this order:
Nonefor a status that cannot change by asking again later: anything but 408, 429, 500, 502, 503 and 504.Nonewhenattempthas reachedtries.- With a
Retry-Afterof whole seconds, that number. - With a
Retry-Afterthat is an HTTP date, the seconds until that date counted from the answer's ownDateheader, or fromnowwhen there is noDate;0when the date has passed. - Otherwise, or when
Retry-Afteris neither, full jitter:rng.uniform(0, min(cap, base * 2 ** (attempt - 1))).
email.utils.parsedate_to_datetime(value).timestamp() turns an HTTP date into epoch seconds and raises ValueError or TypeError for a value that is not one. The sample tests cover every rule, the clock of a host that runs 40 s behind yours among them.
Starter code · retrydelay.py
import email.utils
RETRYABLE = {408, 429, 500, 502, 503, 504}
def retry_delay(status, headers, attempt, now, rng, base=2.0, cap=30.0, tries=3):
"""Seconds to wait before trying a failed GET again, or None when it should not be tried again now."""
# Replace this: a client that retries everything after one second hammers a host that asked it to wait.
return 1.0The sample tests · test_retrydelay.py
import random
from email.utils import formatdate
from retrydelay import retry_delay
NOW = 1_800_000_000 # your clock, in epoch seconds
def test_answers_that_cannot_change_are_not_retried():
"""only 408, 429 and the 5xx answers 500, 502, 503 and 504 can change by asking later"""
for status in (400, 401, 403, 404, 410, 501):
assert retry_delay(status, {}, 1, NOW, random.Random(1)) is None, f"{status} will not change by asking again"
def test_no_retry_once_the_tries_are_used_up():
"""attempt 3 of 3 failed: give up for now"""
assert retry_delay(503, {}, 3, NOW, random.Random(1)) is None, "the third of three tries just failed"
assert retry_delay(503, {}, 2, NOW, random.Random(1)) is not None, "try 2 of 3 failed: one try is left"
assert retry_delay(503, {}, 4, NOW, random.Random(1), tries=5) is not None, "tries says how many a request gets"
def test_retry_after_in_seconds():
"""Retry-After: 30 means wait 30 s, whatever the clocks say"""
assert retry_delay(429, {"Retry-After": "30"}, 1, NOW, random.Random(1)) == 30
assert retry_delay(503, {"retry-after": " 120 "}, 1, NOW, random.Random(1)) == 120, "header names come in any case"
def test_retry_after_date_counted_from_the_answers_date():
"""the host's clock is 40 s behind yours: count from its Date header, not from now"""
headers = {"Date": formatdate(NOW - 40, usegmt=True), "Retry-After": formatdate(NOW + 50, usegmt=True)}
assert retry_delay(503, headers, 1, NOW, random.Random(1)) == 90, "from its Date to its Retry-After is 90 s"
no_date = {"Retry-After": formatdate(NOW + 50, usegmt=True)}
assert retry_delay(503, no_date, 1, NOW, random.Random(1)) == 50, "without a Date header, count from now"
past = {"Retry-After": formatdate(NOW - 10, usegmt=True)}
assert retry_delay(503, past, 1, NOW, random.Random(1)) == 0, "a date that has passed: no wait"
def test_full_jitter_without_retry_after():
"""no usable Retry-After: a random wait from 0 up to a window that doubles and stops at the cap"""
assert retry_delay(503, {}, 1, NOW, random.Random(5)) == random.Random(5).uniform(0, 2.0), "window 2 s"
assert retry_delay(500, {}, 3, NOW, random.Random(5), tries=9) == random.Random(5).uniform(0, 8.0), "window 8 s"
assert retry_delay(502, {}, 8, NOW, random.Random(5), tries=9) == random.Random(5).uniform(0, 30.0), "capped at 30 s"
assert retry_delay(504, {"Retry-After": "soon"}, 1, NOW, random.Random(5)) == random.Random(5).uniform(0, 2.0), \
"a Retry-After that is neither seconds nor a date counts as missing"A hint
Lower-case the header names first, then follow the rules in their order and return as soon as one applies. For Retry-After, value.strip().isdigit() tells whole seconds from a date; put the date parsing inside try and fall through to the jitter when it fails.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Exercise · Medium · Python
A client that paces itself and retries within limits
Finish PoliteClient in politeclient.py. It fetches pages from one host through send(path), which returns (status, headers), on a clock with now() and sleep(seconds). get(path) returns the last status it received.
- Pace every request, retries included, with a token bucket of your own:
burstrequests may start at once, thenratea second. Before each request, sleep as long as the bucket needs to hold a token, then spend it. - **Retry with
delay**: after an answer that is not2xx, calldelay(status, headers, attempt), whereattemptis the try that just failed (1for the first). It returns the seconds to wait, orNoneto stop trying. Sleep that long, then try again, paced like any other request. - **Stop after
triestries** of one path, whateverdelaysays.
The starter's simple_delay is a stand-in for the retry_delay of the lab's first step: Retry-After in seconds, else 1 s, for 408, 429 and the 5xx answers, and None for every other status. The sample tests give the client a host whose clock never waits, and check the moment every request reached it.
Starter code · politeclient.py
def simple_delay(status, headers, attempt):
"""A stand-in for retry_delay: Retry-After in seconds, else 1 s, for answers that may change; None for the rest."""
if status not in (408, 429, 500, 502, 503, 504):
return None
value = {k.lower(): v for k, v in headers.items()}.get("retry-after", "").strip()
return int(value) if value.isdigit() else 1.0
class PoliteClient:
def __init__(self, send, clock, rate=1.0, burst=2, tries=3, delay=simple_delay):
self.send, self.clock, self.rate, self.burst, self.tries, self.delay = send, clock, rate, burst, tries, delay
def get(self, path):
"""Fetch path, pacing every request with the bucket and retrying with delay; return the last status."""
# Replace this: one request with no pacing and no retry.
status, headers = self.send(path)
return statusThe sample tests · test_politeclient.py
from politeclient import PoliteClient
class Clock:
def __init__(self):
self.t = 0.0
def now(self):
return self.t
def sleep(self, seconds):
assert seconds >= 0, "sleep() was asked to wait a negative time"
self.t += seconds
class Host:
"""Answers each path from a script of (status, headers) pairs (the last one repeats) and logs when each came."""
def __init__(self, clock, scripts):
self.clock, self.scripts, self.log = clock, scripts, []
def send(self, path):
self.log.append((round(self.clock.now(), 6), path))
script = self.scripts.get(path, [(200, {})])
n = sum(1 for _, p in self.log if p == path)
return script[min(n, len(script)) - 1]
def setup(scripts=None, **options):
clock = Clock()
host = Host(clock, scripts or {})
return clock, host, PoliteClient(host.send, clock, **options)
def test_a_burst_then_the_rate():
"""burst=2, rate=1: two requests at once, then one a second"""
clock, host, client = setup(rate=1.0, burst=2)
for page in range(5):
assert client.get(f"/p{page}") == 200
assert [t for t, _ in host.log] == [0, 0, 1, 2, 3], "the third request waits for a token"
def test_the_bucket_refills_while_idle():
"""an idle client earns tokens again, never more than burst"""
clock, host, client = setup(rate=0.5, burst=2)
client.get("/a")
client.get("/b")
clock.sleep(100)
for page in "cde":
client.get(f"/{page}")
assert [t for t, _ in host.log] == [0, 0, 100, 100, 102], "100 s idle refills the 2 tokens, no more"
def test_a_part_refilled_bucket_waits_for_the_rest():
"""rate=0.5: after 1 s idle the bucket holds half a token, so /c waits 1 s and /d 2 s more"""
clock, host, client = setup(rate=0.5, burst=2)
client.get("/a")
client.get("/b")
clock.sleep(1)
client.get("/c")
client.get("/d")
assert [t for t, _ in host.log] == [0, 0, 2, 4], "half a token is not a token: /c waits until 2 s, /d until 4 s"
def test_retry_after_is_obeyed():
"""429 with Retry-After: 5, then 200: the second try comes 5 s later"""
clock, host, client = setup({"/x": [(429, {"Retry-After": "5"}), (200, {})]}, rate=1.0, burst=2)
assert client.get("/x") == 200
assert [t for t, _ in host.log] == [0, 5], "nothing may reach the host inside the 5 s it asked for"
def test_retries_are_paced_too():
"""a retry spends a token like any other request"""
clock, host, client = setup({"/y": [(503, {}), (503, {}), (200, {})]}, rate=0.25, burst=1)
assert client.get("/y") == 200
assert [t for t, _ in host.log] == [0, 4, 8], "rate 0.25 means 4 s between starts, even after a 1 s delay"
def test_final_answers_are_not_retried():
"""404 and 403 do not change by asking again"""
clock, host, client = setup({"/gone": [(404, {})], "/no": [(403, {})]})
assert client.get("/gone") == 404
assert client.get("/no") == 403
assert len(host.log) == 2, "one request each"
def test_a_success_is_never_retried():
"""a 2xx ends the request, even with a delay function that would retry anything"""
clock, host, client = setup({"/ok": [(204, {})]}, delay=lambda status, headers, attempt: 1.0)
assert client.get("/ok") == 204
assert len(host.log) == 1, "delay is asked only after an answer that is not 2xx"
def test_tries_cap_the_retries():
"""503 forever: three tries, then the last status is returned"""
clock, host, client = setup({"/z": [(503, {})]}, tries=3)
assert client.get("/z") == 503
assert len(host.log) == 3, "tries=3 means at most three requests for one path"A hint
Keep the bucket in two numbers, the tokens and the time you last counted them. Before a request, add (now - then) * rate tokens (never more than burst); if fewer than 1 are left, sleep (1 - tokens) / rate and count again. A loop of for attempt in range(1, self.tries + 1): around one request makes the third rule easy.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
The practice session starts with the idea your checkpoint answers left weakest:
Practice that adapts to you
Practise this lesson
14 questions about 5 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What does HTTP 429 mean, and what should a collector do when it gets one?
429 Too Many Requests, defined in RFC 6585, says the client has asked more often in a stretch of time than the
server is willing to answer. It is about your pace, not about the page, so retrying at once only adds refusals.
Read Retry-After if the answer has one, as a number of seconds or as an HTTP date counted from the answer’s own
Date header, and send nothing to that host until then; without it, wait with jittered exponential backoff. Then
slow the collector down, for example with a token bucket a margin below the limit, because a well-paced collector
never sees a 429. Retry only safe requests such as GET, a few times, and log every 429 as a sign the pace needs
fixing.
Two harder questions grow out of this lesson: a nightly job of 30,000 prices that fails on some nights, and a partner’s outage that your retries made worse. The question page gives each one’s first step and the follow-ups an interviewer would ask:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
A nightly job collects 30,000 prices from a partner's API, which allows 10 requests a second per key. Some nights it finishes in under two hours; on others it stops with hundreds of 429 errors. How do you make it finish every night, and how long should it take?
-
Mid to senior
During a partner's outage, their API saw your collector's traffic jump to about four times its normal rate, and they asked you to stop. Your client retries every failed request three times at once, and four workers run it. What went wrong, and what do you change?
Key takeaways
- A rate limit is a budget, often a token bucket: a burst passes and then a steady rate, and a 429 with
Retry-Aftersays the budget is spent; in the lab, 20 requests in a row got 5 answers and 15 refusals. - Pace below the limit with a fixed pause plus jitter or with a token bucket of your own a margin under the server’s; both sent exactly 20 requests for 20 pages, against 301 for a client that asked again at once.
- Wait the full
Retry-After: count a number of seconds from the answer and a date from the answer’s ownDateheader, because a clock 40 s off turned 2 requests into 41. - Without
Retry-After, back off exponentially with full jitter and a cap: it sent 168 requests where waits without jitter sent 275 in waves, and every collector was served by 19.5 s instead of 180.5 s. - Retry only 408, 429 and the temporary 5xx answers, only for safe methods such as GET, a few times each and within a run-wide budget; in an outage the budget held 100 failing pages to 110 requests instead of 300.
Terms in this lesson
- exponential backoff
- Waiting longer after each failure before trying again, the wait doubling each time up to a cap, such as 2, 4, 8 and 16 s up to 30 s.
- idempotent method
- A request method whose intended effect is the same whether a request is sent once or several times, such as GET, HEAD, PUT and DELETE in RFC 9110, so a client may repeat it after a failure. POST is not one.
- jitter
- A random part added to a wait, such as a pause of 1.1 s varied by up to 10 %, so that clients started at the same moment stop sending their requests in step.
- retry budget
- A limit on retries for a whole run or client rather than per request, such as retries no more than 10 % of first tries, so that a failing host gets little extra load.
- token bucket
- A way to limit a rate: a bucket holds up to a set number of tokens, each request spends one, and tokens come back at a steady rate, so a short burst passes and then a steady pace.
References
- RFC 6585: Additional HTTP Status Codes (section 4, 429 Too Many Requests; section 7.2) (IETF)
- RFC 9110: HTTP Semantics (sections 6.6.1, 9.2.2, 10.2.3, 15.5.9 and 15.6) (IETF)
- RFC 9111: HTTP Caching (section 4.2.1, freshness lifetime and clock skew) (IETF)
- urllib3 2.8: urllib3.util.Retry (urllib3 project)
- Tenacity: API reference (wait and stop strategies) (Tenacity project)
- Tenacity: Retrying library for Python (default behaviour) (Tenacity project)
- Exponential Backoff And Jitter (Amazon Web Services (AWS Architecture Blog))
- Timeouts, retries and backoff with jitter (Amazon Web Services (Amazon Builders' Library))
- Site Reliability Engineering, chapter 21: Handling Overload (Google (O'Reilly Media))
- Site Reliability Engineering, chapter 22: Addressing Cascading Failures (Google (O'Reilly Media))
- Rate limits for the REST API (secondary rate limits) (GitHub)
- Scrapy 2.19: Downloader middleware (RETRY_HTTP_CODES, RETRY_TIMES) (Scrapy developers)
- Scrapy 2.19: Settings (DOWNLOAD_DELAY, DOWNLOAD_DELAY_JITTER) (Scrapy developers)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress