Python Module 7 – Errors, exceptions and context managers
Designing error handling for real programs
Where to catch, log, convert or re-raise errors in Python, how to show users safe messages and exit statuses, and how to retry temporary failures.
What you will learn
- Decide where to catch, log, convert or re-raise errors
- Write safe, actionable error messages and return correct exit statuses
- Log uncaught exceptions with sys.excepthook and threading.excepthook
- Add retries with backoff and jitter for temporary failures
Before you start
What you’ll learn · Before you start
What you will learn
- Decide where to catch, log, convert or re-raise errors
- Write safe, actionable error messages and return correct exit statuses
- Log uncaught exceptions with sys.excepthook and threading.excepthook
- Add retries with backoff and jitter for temporary failures
Before you start
On this page
A helper that catches an error, prints a line and carries on can turn one bad price out of three into a crash in
another function that no longer names the price, or into a wrong total with no error at all. Good error handling is
mostly a decision about where each error is dealt with. Helpers raise and let the error travel. At each
module boundary, a low-level error such as ValueError becomes a custom exception: An exception class a program defines for its own errors, derived from Exception, such as SeatTakenError. It can keep the details of the error as attributes for the code that handles it. More in “Raising exceptions and custom error classes” of your own domain, such as OrderError, chained to the original. At the program’s top-level
boundary, such as main() or a request handler, the error is caught: the details go to a log, the user gets a
short message that says what to do, and the program returns an exit status: The small number a program hands back to whatever started it: 0 for success, anything else for failure. Shells, CI jobs and schedulers read it; by convention 2 means a wrong command line and 1 any other error. or an
error response.
Below you will build each layer in examples you can run, add two hooks for what slips past them, and retry temporary failures with waits measured on 1,000 clients that fail together.
Most functions should let an error travel
When an exception reaches your code, there are four reasonable choices:
- Let it propagate. Most functions should not catch anything. If a function cannot do anything useful about an error, the error belongs to its caller.
- Add context and pass it on.
e.add_note("while reading orders.csv")adds a line to the traceback, andraise OrderError(...) from ereplaces a technical error with one the caller understands, keeping the original as __cause__: The earlier exception named by raise ... from ..., the stated reason for the new one. The traceback prints it first, followed by "The above exception was the direct cause of the following exception". More in “Exception chaining, notes and tracebacks”. - Handle it. Recover for real: use a default, skip a bad record and count it, retry a call that failed for a temporary reason.
- Report it at the boundary. Log the details, tell the user what happened and what to do, and set the exit status or the response.
The tempting fifth choice, catching an error deep inside a helper, printing something and carrying on, is the one to
avoid. Here read_price() does exactly that:
def read_price(text):
try:
return float(text)
except ValueError:
print("could not read price") # handled too early: the caller cannot tell
return None
def order_total(lines):
return sum(read_price(price) * int(qty) for price, qty in lines)
try:
print(order_total([("12.50", "2"), ("abc", "1"), ("4.00", "3")]))
except Exception as e:
print(f"the caller sees {type(e).__name__}: {e}")Output
could not read price the caller sees TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 swallowed.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The real problem is the price "abc". Step through the three order lines and watch what happens to its ValueError:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Three order lines. The caller passes three lines to order_total(), each a price and a quantity, ("12.50", "2"), ("abc", "1") and ("4.00", "3"), inside a try statement.
- A good price. order_total() asks read_price() for the first price.
- 12.5 times 2. float("12.50") works, so read_price() returns 12.5, and order_total() multiplies it by the quantity, 2.
- The bad price. For the second line, read_price() gets the text "abc".
- Caught too early. float("abc") raises ValueError, whose message names the bad value. read_price() catches it, prints "could not read price" and returns None, and the ValueError is gone.
- None travels on. order_total() cannot tell None from a price, so it multiplies None by the quantity, 1.
- A different error. None * 1 raises TypeError: unsupported operand type(s) for *: 'NoneType' and 'int', in a function that did nothing wrong.
- What the caller sees. The TypeError reaches the caller, which prints it. Nothing in it names the value "abc" or its line, so finding the real problem now means reading the code.
Because read_price() swallowed the ValueError and returned None, the error that finally escapes is a
TypeError about NoneType, raised in another function, and it no longer says which value was bad. A caught
exception that is not really handled turns one clear error into a later, confusing one, or into a wrong total with no
error at all.
Key idea
Catch an error only where you can do something about it; everywhere else, let it travel, adding context on the way if you know something the error does not.
Checkpoint
Check: What to do with an error that reaches your code
A module turns low-level errors into one error of its own
A module’s callers should not need to know how it works inside. The diagram shows the three layers an error passes on its way up, and what each layer does with it:
Where each error is handled, from the helpers up to the boundary
Text description of the diagram
The diagram shows three layers of a program, with errors travelling upwards.
- At the bottom are the helpers, such as open(), float() or a network call. They raise exceptions and may add context with add_note(), but they never print a message and carry on as if nothing had happened.
- In the middle is a module boundary, such as a function that loads orders or a client for an API. Low-level errors such as FileNotFoundError, ValueError or TimeoutError arrive here and are converted into an error of the program's domain, for example raise OrderError(...) from e, which keeps the original as the cause. Temporary failures are retried at this layer.
- At the top is the program's boundary: the main() function of a command-line tool, a request handler or a job runner. The domain error, with its cause chained, arrives here. This layer catches it, logs the details for developers, shows the user a short message, and returns an exit status or an error response.
A function that loads orders may fail because a file is missing, a number does not parse or a column is absent,
which raise FileNotFoundError, ValueError and KeyError: three exception types for one kind of failure.
Callers want one answer to one question, “did loading the orders fail, and how do I tell the user?”. So the module
catches the low-level errors it expects at its edge and raises its own exception, with a message written for the
person who will read it:
Python (try_cli.py) · cli/try_cli.py
import sys
from orders_cli import main
sys.stderr = sys.stdout # show error messages in the same output, in order
for argv in (["orders.csv"], [], ["missing.csv"], ["bad_orders.csv"]):
print("$", " ".join(["python3 orders_cli.py", *argv]))
print("exit status", main(argv))Output
$ python3 orders_cli.py orders.csv 3 orders, total 1155.50 exit status 0 $ python3 orders_cli.py usage: orders_cli.py FILE exit status 2 $ python3 orders_cli.py missing.csv error: missing.csv does not exist; check the name and the folder exit status 1 $ python3 orders_cli.py bad_orders.csv error: bad_orders.csv, line 3: price and qty must be numbers exit status 1
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 try_cli.py
Python (orders_cli.py) · cli/orders_cli.py
"""Total the orders in a CSV file: python3 orders_cli.py FILE"""
import csv
import sys
class OrderError(Exception):
"""A problem the user can fix; the message says what to do."""
def load_orders(path):
try:
with open(path, newline="", encoding="utf-8") as f:
rows = list(csv.DictReader(f))
except FileNotFoundError as e:
raise OrderError(f"{path} does not exist; check the name and the folder") from e
orders = []
for line, row in enumerate(rows, start=2): # line 1 holds the column names
try:
orders.append((float(row["price"]), int(row["qty"])))
except (KeyError, TypeError, ValueError) as e:
raise OrderError(f"{path}, line {line}: price and qty must be numbers") from e
return orders
def main(argv):
"""Run the command and return its exit status: 0 success, 1 error, 2 wrong usage."""
if len(argv) != 1:
print("usage: orders_cli.py FILE", file=sys.stderr)
return 2
try:
orders = load_orders(argv[0])
except OrderError as e:
print(f"error: {e}", file=sys.stderr)
return 1
print(f"{len(orders)} orders, total {sum(price * qty for price, qty in orders):.2f}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))CSV (orders.csv) · cli/orders.csv
item,price,qty
notebook,45.50,3
pen,12.00,10
bag,899.00,1CSV (bad_orders.csv) · cli/bad_orders.csv
item,price,qty
notebook,45.50,3
pen,abc,10Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
In orders_cli.py, load_orders() turns FileNotFoundError into OrderError("missing.csv does not exist; check the name and the folder"), and a bad number into an OrderError that names the file and the line. raise ... from e
keeps the original exception as __cause__, so a log or a traceback still shows the ValueError underneath, while
the message stays readable. main() then needs one except OrderError clause: everything the user can fix arrives
as an OrderError, and anything else is a bug in the program. try_cli.py calls main() with four lists of
arguments, the way four separate runs from a terminal would, and shows error output in the same stream so the order
is clear. Step through the last of them, the file with the bad price, across the three layers of the diagram:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- One argument. try_cli.py calls main(["bad_orders.csv"]). There is exactly one argument, so main() calls load_orders() with it.
- Line 2 converts. The file opens and csv.DictReader reads its rows. Line 2, the notebook, converts to a price of 45.5 and a quantity of 3.
- Line 3 does not. Line 3, the pen, has the price "abc". float("abc") raises ValueError: could not convert string to float: 'abc'.
- Caught at the edge. The ValueError reaches the except clause of load_orders(), which names KeyError, TypeError and ValueError, the ways a row can be bad.
- Converted, cause kept. load_orders() raises OrderError("bad_orders.csv, line 3: price and qty must be numbers") from e. The message is written for the user; __cause__ keeps the ValueError for a log or a traceback.
- One clause in main(). The OrderError reaches main(), which needs a single except OrderError clause, because everything the user can fix arrives as an OrderError.
- A message and a status. main() prints "error: bad_orders.csv, line 3: price and qty must be numbers" to standard error and returns 1, the status sys.exit() hands to the shell when the tool runs for real.
Notice what main() does not catch. A bug, such as a typo in a variable name, is not an OrderError, so it
escapes with its full traceback. That is deliberate: a short, friendly message for a bug would hide the one piece of
information that lets someone fix it. When you do need a catch-all, for example in a server that must keep serving,
catch Exception, never BaseException or a bare except:, so that Ctrl-C (KeyboardInterrupt) and sys.exit()
(SystemExit) still work, and log the traceback.
Checkpoint
Check: Converting errors at a module boundary
Exit statuses tell the shell what happened
A program tells the shell, a CI job or another program whether it worked through its exit status, a small number.
0 means success and anything else failure. The documentation of sys.exit() notes the Unix habit of 2 for a
wrong command line and 1 for every other error, and argparse exits with 2 when it rejects the arguments.
Statuses should stay between 0 and 127. Write main(argv) so that it returns the status, and finish the file with
sys.exit(main(sys.argv[1:])): tests can call main() with any arguments and check the number, and the shell gets
the same number when the program runs for real. These are the statuses a Python program ends with, and where each
one comes from:
| What happened | Exit status | Where the number comes from |
|---|---|---|
| The command did its job | 0 | main() returns 0 |
| The command line was wrong | 2 | main() returns 2, as argparse does |
| An error the user can fix | 1 | main() prints the message and returns 1 |
sys.exit("message") |
1 | Python prints the message to standard error |
| An exception that nothing caught, Ctrl-C aside | 1 | Python prints the traceback first |
The four runs of try_cli.py above return 0, 2, 1 and 1: the first three rows. Ctrl-C is the one exception to the
last row: since Python 3.8, a KeyboardInterrupt that nothing caught ends the program through the interrupt signal
itself, so the shell or the program that started it can tell that the run was interrupted. sys.exit() works by
raising SystemExit, so finally clauses and with blocks still clean up on the way out. A message for the user
names the problem in their terms and the next step, such as error: bad_orders.csv, line 3: price and qty must be numbers, and goes to standard error, so that it never mixes with output that another program may read from standard
output.
Checkpoint
Check: Exit statuses of a command-line program
Users get a sentence, the log gets the traceback
A job that runs unattended, such as a nightly report, has two audiences: the person told that it failed, and the developer who has to work out why. They need different things:
import io
import logging
log_file = io.StringIO() # stands in for a log file or a log service
logging.basicConfig(stream=log_file, level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("reports")
RATES = {"standard": 0.18, "reduced": 0.05}
def tax(amount, rate_name):
return round(amount * RATES[rate_name], 2)
def build_report(lines):
return [(item, amount, tax(amount, rate)) for item, amount, rate in lines]
def run_job(lines):
try:
report = build_report(lines)
except Exception:
log.exception("report failed for %d lines", len(lines))
print("The report could not be built, and nothing was sent. Try again later; if it keeps failing, contact support.")
return 1
log.info("report built with %d lines", len(report))
print("Report ready:", report)
return 0
run_job([("notebook", 100.0, "standard"), ("rice", 250.0, "reduced")])
run_job([("notebook", 100.0, "standard"), ("rice", 250.0, "reduced"), ("milk", 60.0, "zero")])
print("--- what the log holds ---")
print(log_file.getvalue(), end="")Output
Report ready: [('notebook', 100.0, 18.0), ('rice', 250.0, 12.5)]
The report could not be built, and nothing was sent. Try again later; if it keeps failing, contact support.
--- what the log holds ---
INFO reports: report built with 2 lines
ERROR reports: report failed for 3 lines
Traceback (most recent call last):
File "logged.py", line 21, in run_job
report = build_report(lines)
File "logged.py", line 16, in build_report
return [(item, amount, tax(amount, rate)) for item, amount, rate in lines]
~~~^^^^^^^^^^^^^^
File "logged.py", line 12, in tax
return round(amount * RATES[rate_name], 2)
~~~~~^^^^^^^^^^^
KeyError: 'zero'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 logged.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The user sees one sentence: what failed, what did not happen (nothing was sent) and what to do. The log gets
logger.exception(), which records the message at the ERROR level together with the full traceback, ending in
KeyError: 'zero', a tax rate the code does not know. logger.exception() only makes sense inside an except
clause, where there is an exception to record. Log an error once, at the layer that handles it; when every layer logs
and re-raises, one failure fills the log with the same traceback several times.
Security
Messages that reach users must not contain tracebacks, file paths, SQL, server or library versions, or secrets: each of those helps an attacker learn how the system is built, which is why OWASP’s error-handling guidance asks for generic responses to unexpected errors and the details in server-side logs. Exception messages often end up in both places, so never put a password, a token or a full connection string into one.
Checkpoint
Check: Messages for people, details for the log
Two hooks report what nothing caught
Whatever no except clause catches in the main thread goes to sys.excepthook(type, value, traceback) just before
the program ends; its default prints the traceback to standard error. An exception that ends a thread goes to
threading.excepthook(args) instead, whose args has exc_type, exc_value, exc_traceback and thread. Both can
be replaced, which is how a program sends every uncaught failure to its log:
import sys
import threading
def report_uncaught(exc_type, exc, tb):
print(f"[log] uncaught {exc_type.__name__}: {exc}", file=sys.stderr)
print("Something went wrong; the details are in the log.", file=sys.stderr)
def report_thread(args):
print(f"[log] thread {args.thread.name} failed: {args.exc_type.__name__}: {args.exc_value}", file=sys.stderr)
sys.excepthook = report_uncaught
threading.excepthook = report_thread
worker = threading.Thread(target=lambda: {}["missing"], name="export-worker")
worker.start()
worker.join()
print("the main thread carries on after the thread failed")
raise RuntimeError("config.toml has no [database] table")Output (exit status 1)
the main thread carries on after the thread failed
Printed as an error (standard error)
[log] thread export-worker failed: KeyError: 'missing' [log] uncaught RuntimeError: config.toml has no [database] table Something went wrong; the details are in the log.
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 last_resort.py
This example starts a thread, which the browser runner cannot do, so the page shows its recorded output. Step through the run and watch which hook each failure reaches:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Two hooks replaced. The program replaces both hooks with its own functions, report_uncaught for sys.excepthook and report_thread for threading.excepthook. Nothing has failed yet.
- A thread starts. The main thread starts a thread called export-worker and waits for it with join(). The thread's only work is {}["missing"], a lookup in an empty dictionary.
- The thread fails. The lookup raises KeyError: 'missing'. Nothing in the thread catches it, so the thread ends.
- The thread's hook. An exception that ends a thread goes to threading.excepthook, not sys.excepthook. report_thread(args) prints "[log] thread export-worker failed: KeyError: 'missing'" from args.thread, args.exc_type and args.exc_value.
- The main thread carries on. join() returns and the main thread prints that it carries on. An exception in one thread never stops the others.
- An uncaught RuntimeError. The main thread raises RuntimeError("config.toml has no [database] table") and nothing catches it. Just before the program ends, Python calls sys.excepthook with its type, value and traceback.
- Reported, not printed raw. report_uncaught prints "[log] uncaught RuntimeError" with the message, then a short sentence for the user, instead of the default traceback.
- Still status 1. The program still ends with exit status 1. The hooks decide how a failure is reported, not whether the program survives it.
The failed thread is reported, and the main thread carries on, because an exception in a thread never stops the
others. The uncaught RuntimeError then goes to the new sys.excepthook, and the program still ends with status 1:
the hooks decide how a failure is reported, not whether the program survives it. SystemExit never reaches
sys.excepthook, and threading.excepthook ignores it silently. A similar hook, sys.unraisablehook, receives
exceptions that Python cannot raise anywhere, such as one inside a destructor.
Checkpoint
Check: The hooks for uncaught exceptions
Retry only temporary failures, with waits that grow
Some failures go away by themselves: a connection reset, a timeout, a service that answers 503 Service Unavailable
while it restarts. Others fail the same way every time until someone changes something: invalid input, a missing
permission, a bug. Only the first kind deserves a retry, and only for operations that are safe to repeat. RFC 9110,
the HTTP specification, calls a method idempotent: Said of an operation whose intended effect is the same whether it runs once or several times, such as an HTTP GET or PUT. Only such an operation, or one the server can recognise as a repeat, is safe to retry automatically. when sending the same request twice has
the same intended effect as sending it once: PUT, DELETE and safe methods such as GET. For any other request it
advises against automatic retries, unless the client has some way to know that a repeat does no harm. A payment sent
twice is two payments, unless the server can recognise the repeat, for example by an idempotency key that the client
sends with both.
A good retry waits longer each time, exponential backoff: Waiting longer before each new attempt, doubling the wait each time up to a cap, so that a failing service gets time to recover instead of a steady stream of retries.: after attempt n fails
it waits base_delay × 2ⁿ⁻¹, capped at a maximum. It gives up after a fixed number of attempts and raises the last
error, and when a server sends a Retry-After header, it waits at least that long. The helper below adds
jitter: A random part of each retry's wait, so that clients that failed at the same moment do not all try again at the same moment., a random part of the wait, and takes the sleep function as a parameter, so a test
can record the waits instead of sleeping:
import random
import time
def retry(fn, attempts=4, base_delay=0.5, max_delay=8.0, retry_on=(ConnectionError, TimeoutError), sleep=time.sleep, rng=None):
"""Call fn(); after a temporary error wait base_delay * 2**n seconds, cap it, add jitter, and try again."""
if attempts < 1:
raise ValueError(f"attempts must be at least 1, got {attempts}")
rng = rng or random.Random()
for attempt in range(1, attempts + 1):
try:
return fn()
except retry_on as e:
if attempt == attempts:
e.add_note(f"gave up after {attempts} attempts")
raise
backoff = min(max_delay, base_delay * 2 ** (attempt - 1))
delay = rng.uniform(0, backoff) # jitter: anywhere from 0 to the backoff
print(f" attempt {attempt} failed ({type(e).__name__}); waiting {delay:.2f} s of up to {backoff:.1f} s")
sleep(delay)
def flaky(failures):
"""A stand-in for a network call that fails `failures` times, then works."""
calls = 0
def call():
nonlocal calls
calls += 1
if calls <= failures:
raise ConnectionError("connection reset by peer")
return f"ok after {calls} calls"
return call
waits = []
rng = random.Random(7) # seeded, so the run is the same every time
print("two failures, then success:")
print(retry(flaky(2), sleep=waits.append, rng=rng))
print("waited", round(sum(waits), 2), "s in all")
print("a mistake in the request is not retried:")
try:
retry(lambda: int("12,50"), sleep=waits.append, rng=rng)
except ValueError as e:
print(" ValueError at once:", e)
print("the service stays down:")
try:
retry(flaky(10), sleep=waits.append, rng=rng)
except ConnectionError as e:
print(" ConnectionError:", e, "|", e.__notes__[0])Output
two failures, then success: attempt 1 failed (ConnectionError); waiting 0.16 s of up to 0.5 s attempt 2 failed (ConnectionError); waiting 0.15 s of up to 1.0 s ok after 3 calls waited 0.31 s in all a mistake in the request is not retried: ValueError at once: invalid literal for int() with base 10: '12,50' the service stays down: attempt 1 failed (ConnectionError); waiting 0.33 s of up to 0.5 s attempt 2 failed (ConnectionError); waiting 0.07 s of up to 1.0 s attempt 3 failed (ConnectionError); waiting 1.07 s of up to 2.0 s ConnectionError: connection reset by peer | gave up after 4 attempts
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 retry.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The random generator is seeded, so the run is the same every time and the waits can be checked. A ValueError is not
in retry_on, so it is raised at once: retrying a request that is wrong only repeats the mistake. When every attempt
fails, the last ConnectionError is raised again with a note saying how many attempts were made. Step through the
first and the last run, and watch the limit of each wait double while the wait itself falls anywhere below it:
The drawing shows the whole scene; its 11 steps are listed below.
Every step as text
- Attempt 1. retry() calls fn(), a stand-in for a network call that fails twice and then works.
- A temporary error. The call fails with ConnectionError, which is in retry_on, so retry() will try again.
- Up to 0.5 s. The backoff for attempt 1 is base_delay × 2⁰ = 0.5 s. Jitter picks a wait anywhere from 0 to that, here 0.16 s, which the example records instead of sleeping.
- Attempt 2. The second attempt fails the same way.
- Up to 1.0 s. The backoff doubles to 1.0 s, and the random wait is 0.15 s. A random wait can be shorter than the one before; only its upper limit grows.
- Attempt 3 works. The third attempt succeeds, and the call returns "ok after 3 calls".
- The result. retry() hands the result to the caller. The two waits came to 0.31 s in all.
- A service that stays down. The last run of the example uses a service that fails ten times, more often than retry() will try.
- Three failures, three waits. Attempts 1, 2 and 3 fail. Their backoffs are 0.5, 1.0 and 2.0 s, and the random waits 0.33, 0.07 and 1.07 s.
- The last attempt. Attempt 4 fails too, and it is the last, so there is no final wait.
- Give up, and say so. retry() adds the note "gave up after 4 attempts" to the ConnectionError and raises it again, so the caller sees the real error and how hard the code tried.
Jitter keeps a crowd of retries apart
Jitter matters when many clients fail at the same moment, for example when a server restarts and every client loses its connection at once. With fixed waits they all come back together, again and again. This simulation counts, for 1,000 clients that each retry four times, the most retries that land in any one 10 ms slot:
import random
from collections import Counter
CLIENTS = 1000
BASE = 0.1 # seconds before the first retry
WINDOW = 0.01 # count retries in 10 ms slots
def busiest_slot(jitter, retries=4, seed=1):
"""1,000 clients fail at the same moment; return the most retries that land in any one 10 ms slot."""
rng = random.Random(seed)
slots = Counter()
for _ in range(CLIENTS):
t = 0.0
for n in range(retries):
backoff = BASE * 2**n
t += jitter(rng, backoff)
slots[int(t / WINDOW)] += 1
return max(slots.values())
plans = {
"no jitter (wait exactly the backoff)": lambda rng, b: b,
"small jitter (backoff +/- 20 %)": lambda rng, b: b * rng.uniform(0.8, 1.2),
"full range (0 to the backoff)": lambda rng, b: rng.uniform(0, b),
}
print(f"{CLIENTS:,} clients fail together, then retry 4 times starting at {BASE} s:")
for name, jitter in plans.items():
print(f" {name:38} busiest 10 ms slot: {busiest_slot(jitter):5,} retries")Output
1,000 clients fail together, then retry 4 times starting at 0.1 s: no jitter (wait exactly the backoff) busiest 10 ms slot: 1,000 retries small jitter (backoff +/- 20 %) busiest 10 ms slot: 262 retries full range (0 to the backoff) busiest 10 ms slot: 155 retries
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 herd.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Without jitter every one of the 1,000 retries of each round lands in the same 10 ms slot, so the restarted server meets the full crowd four times. A spread of 20 % either way, the jitter of the gRPC connection backoff protocol, brings the busiest slot down to 262. A wait anywhere between 0 and the backoff brings it down to 155, and also gets clients back sooner on average. One more rule from the same arithmetic: retry in one layer only. If a request handler, the client library it uses and the HTTP layer below each make 3 attempts, one user action sends 27 requests to a service that is already failing.
HTTP Status Codes Reference Look up which HTTP statuses suggest a retry may help, such as 429 and 503, and what Retry-After says.Key idea
Retry only temporary failures of operations that are safe to repeat, in one layer, with waits that double up to a cap, a random part that keeps clients apart, and a limit after which the last error is raised.
Checkpoint
Check: Retries with backoff and jitter
Practise: retries with backoff and a command that sets its exit status
The lesson comes down to a checklist for a real program:
- Each module raises errors of its own domain from a small hierarchy (one base class), chained to the original.
- Exactly one place catches each kind of error; everything else lets it propagate.
main()returns 0, 1 or 2, and the file ends withsys.exit(main(sys.argv[1:])).- Users get short messages on standard error, without tracebacks, paths or secrets; the log gets the traceback, once.
- Uncaught exceptions in the main thread and in threads are logged through
sys.excepthookandthreading.excepthook. - Only temporary failures of idempotent operations are retried, in one layer, with backoff, jitter, a cap and a limit.
- Clean-up uses
withstatements, so every path through the program releases what it acquired.
The exercise practises two lines of it: a retry() whose waits double, and a command that turns each outcome into
the right message and exit status. Its sample tests pass in fake functions, so nothing really waits or connects:
Exercise · Medium · Python
Retries with backoff and a command that sets its exit status
Complete the two functions in stock_cli.py, a small command that looks up how many items a warehouse service has in stock. Keep the names and parameters: the tests call them with fake functions, so nothing really waits or connects.
retry(fn, attempts=3, base_delay=0.1, retry_on=(ConnectionError, TimeoutError), sleep=time.sleep) calls fn() and returns its result.
- When
fn()raises one of the exceptions inretry_on, wait withsleep(delay)and call it again. The waits double:base_delay, then twice that, then four times, and so on. With the defaults the waits are 0.1 s and then 0.2 s. - After
attemptscalls have failed, raise the last exception again, without a final wait. - Any other exception is raised at once, without a retry.
attemptsbelow 1 is a mistake by the caller: raiseValueErrorwithout callingfn().
cli_main(argv, fetch, sleep=time.sleep) runs the command and returns its exit status. argv is the list of arguments, and fetch(item) is the call to the service: it returns the stock as a number, or raises LookupError for an unknown item, or ConnectionError or TimeoutError when the service does not answer.
- Not exactly one argument: print
usage: stock ITEMto standard error and return 2. - Fetch through
retry(), with 3 attempts, a base delay of 0.5 s and thesleepyou were given. - Success: print
<item>: <stock>to standard output, such aspen: 40, and return 0. - An unknown item: print
error: no item called <item>to standard error and return 1, without retrying. - The service still not answering after the retries: print
error: the stock service is not answering; try again in a few minutesto standard error and return 1. - Any other exception is a bug: let it propagate.
Starter code · stock_cli.py
import sys
import time
def retry(fn, attempts=3, base_delay=0.1, retry_on=(ConnectionError, TimeoutError), sleep=time.sleep):
"""Call fn() and return its result, retrying temporary errors with waits that double each time."""
return fn()
def cli_main(argv, fetch, sleep=time.sleep):
"""Print the stock of the item named in argv and return the exit status: 0 success, 1 error, 2 usage."""
print(f"{argv[0]}: {fetch(argv[0])}")
return 0The sample tests · test_stock_cli.py
import io
from contextlib import redirect_stderr, redirect_stdout
from stock_cli import cli_main, retry
def failing(times, error, value="ok"):
"""A fake call that raises `error` for the first `times` calls, then returns `value`."""
calls = []
def call():
calls.append(1)
if len(calls) <= times:
raise error
return value
return call, calls
def run_cli(argv, fetch):
out, err, waits = io.StringIO(), io.StringIO(), []
with redirect_stdout(out), redirect_stderr(err):
status = cli_main(argv, fetch, sleep=waits.append)
return status, out.getvalue(), err.getvalue(), waits
def test_retry_success_needs_no_wait():
"""returns the result of a call that works, without waiting"""
call, calls = failing(0, ConnectionError("down"))
waits = []
assert retry(call, sleep=waits.append) == "ok"
assert waits == []
assert len(calls) == 1
def test_retry_waits_double():
"""retries a temporary error, waiting base_delay, then twice as long"""
call, calls = failing(2, ConnectionError("reset"))
waits = []
assert retry(call, attempts=3, base_delay=0.1, sleep=waits.append) == "ok"
assert waits == [0.1, 0.2]
assert len(calls) == 3
def test_retry_gives_up():
"""raises the last error after the last attempt, without a final wait"""
call, calls = failing(9, TimeoutError("no answer"))
waits = []
try:
retry(call, attempts=4, base_delay=1, sleep=waits.append)
except TimeoutError:
pass
else:
assert False, "retry() did not raise TimeoutError"
assert waits == [1, 2, 4]
assert len(calls) == 4
def test_retry_other_errors_at_once():
"""raises an error that is not in retry_on at once"""
call, calls = failing(5, ValueError("bad request"))
waits = []
try:
retry(call, sleep=waits.append)
except ValueError:
pass
else:
assert False, "retry() did not raise ValueError"
assert waits == []
assert len(calls) == 1
def test_retry_needs_one_attempt():
"""rejects attempts below 1 without calling fn()"""
call, calls = failing(0, ConnectionError("down"))
try:
retry(call, attempts=0, sleep=[].append)
except ValueError:
pass
else:
assert False, "retry(attempts=0) did not raise ValueError"
assert calls == []
def test_cli_success():
"""prints the stock and returns 0"""
status, out, err, waits = run_cli(["pen"], lambda item: 40)
assert status == 0
assert out == "pen: 40\n"
assert err == ""
def test_cli_usage():
"""returns 2 with a usage line when the argument is missing"""
status, out, err, waits = run_cli([], lambda item: 40)
assert status == 2
assert "usage: stock ITEM" in err
assert out == ""
def test_cli_unknown_item():
"""returns 1 for an unknown item, without retrying"""
def fetch(item):
raise LookupError(item)
status, out, err, waits = run_cli(["rubber"], fetch)
assert status == 1
assert err == "error: no item called rubber\n"
assert waits == []
def test_cli_service_down():
"""returns 1 after 3 attempts with waits of 0.5 s and 1 s"""
call, calls = failing(9, ConnectionError("refused"))
status, out, err, waits = run_cli(["pen"], lambda item: call())
assert status == 1
assert err == "error: the stock service is not answering; try again in a few minutes\n"
assert waits == [0.5, 1.0]
assert len(calls) == 3
def test_cli_bug_propagates():
"""lets an unexpected exception through instead of hiding it"""
def fetch(item):
return 1 / 0
try:
run_cli(["pen"], fetch)
except ZeroDivisionError:
return
assert False, "cli_main() hid a ZeroDivisionError"A hint
In retry(), first reject attempts < 1 with ValueError, then loop with for attempt in range(1, attempts + 1): and put return fn() in a try whose except retry_on: either re-raises with a bare raise when attempt == attempts or calls sleep(base_delay * 2 ** (attempt - 1)). Because except names only retry_on, every other exception passes straight through. In cli_main(), check len(argv) first, then call retry(lambda: fetch(item), attempts=3, base_delay=0.5, sleep=sleep) inside a try with one except for LookupError and one for (ConnectionError, TimeoutError). Print errors with print(..., file=sys.stderr).
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then test the ideas of this lesson one question at a time. The session starts where your answers are weakest, and after every choice it tells you what that choice got right or missed:
Practice that adapts to you
Practise this lesson
19 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Fresher to 2–5 years · What exit status should a command-line program return, and why does it matter?
Zero when it succeeded and a non-zero value when it failed, because that number is how shells, CI pipelines,
schedulers and other programs decide whether a step worked and whether to run the next one. By the Unix convention
that Python’s sys.exit() documentation repeats, 2 means the command line was wrong, which is also what argparse
uses, and 1 covers other errors; values should stay between 0 and 127. In Python, write main(argv) to return the
status and call sys.exit(main(sys.argv[1:])). An uncaught exception other than Ctrl-C ends the program with status 1,
so even a crash reports failure. The classic bug is the opposite: catching every error, printing a message and
returning normally, so the program exits with 0 and the automation around it believes it worked.
Key takeaways
- Most code lets errors propagate; modules convert low-level errors into domain errors with
raise ... from e, and the top-level boundary catches, logs and reports. - Never catch, print and carry on inside a helper: it replaces a clear error with a confusing one or a wrong result.
main(argv)returns the exit status, 0 for success, 2 for a wrong command line and 1 for other errors, and the file ends withsys.exit(main(sys.argv[1:])).- Users get short, actionable messages on standard error with no tracebacks, paths or secrets, and
logger.exception()puts the traceback in the log, once. sys.excepthookandthreading.excepthookreport what nothing caught, but they cannot keep the program alive.- Retry only temporary failures of idempotent operations, in one layer, with exponential backoff, jitter, a cap and a
limit, and respect
Retry-After.
Terms in this lesson
- exit status
- The small number a program hands back to whatever started it: 0 for success, anything else for failure. Shells, CI jobs and schedulers read it; by convention 2 means a wrong command line and 1 any other error.
- exponential backoff
- Waiting longer before each new attempt, doubling the wait each time up to a cap, so that a failing service gets time to recover instead of a steady stream of retries.
- idempotent
- Said of an operation whose intended effect is the same whether it runs once or several times, such as an HTTP GET or PUT. Only such an operation, or one the server can recognise as a repeat, is safe to retry automatically.
- jitter
- A random part of each retry's wait, so that clients that failed at the same moment do not all try again at the same moment.
References
- The Python Tutorial: errors and exceptions (Python Software Foundation)
- sys.exit() (Python Software Foundation)
- sys.excepthook() (Python Software Foundation)
- threading.excepthook() (Python Software Foundation)
- logging: Logger.exception() (Python Software Foundation)
- argparse: ArgumentParser.error() (Python Software Foundation)
- Built-in exceptions: SystemExit (Python Software Foundation)
- What's New In Python 3.8: other language changes (Ctrl-C and the exit status) (Python Software Foundation)
- RFC 9110, HTTP Semantics: 9.2.2 Idempotent Methods (IETF)
- RFC 9110, HTTP Semantics: 10.2.3 Retry-After (IETF)
- gRPC Connection Backoff Protocol (The gRPC Authors)
- Error Handling Cheat Sheet (OWASP Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress