Why and how to test Python code

An online shop charges 40 rupees for delivery and waives the fee from 499 rupees up. Write that rule with one wrong character, > where it says “from 499”, and every order of exactly 499 pays the fee, with no error anywhere. A test is a small program that notices: it runs your code with an input you chose, such as 499, and checks that the result is the one you expected. In Python a test can be as plain as a function holding an assert statement.

Code keeps changing, and every fix, new feature or clean-up can break something that used to work; a suite of tests notices within seconds, long before a user does. Below you can step through tests as a tiny runner runs them, measure why most tests should be small, watch a test fail for the wrong reason and pick the inputs that find bugs. The last section reshapes code that reads the clock, so that it can be tested on any day.

A test runs your code and checks one result

Here is the delivery rule with three tests and a short loop that runs them:

"""Three tests for a delivery-fee rule, and a loop that runs them."""


def delivery_fee(order_rupees):
    """Return the delivery fee in rupees: 40 below ₹499, nothing from ₹499 up."""
    if order_rupees < 0:
        raise ValueError(f"an order total cannot be negative: {order_rupees}")
    return 0 if order_rupees >= 499 else 40


def test_small_order_pays_the_fee():
    assert delivery_fee(120) == 40


def test_fee_is_waived_from_499():
    assert delivery_fee(499) == 0
    assert delivery_fee(5000) == 0


def test_negative_total_is_refused():
    try:
        delivery_fee(-1)
    except ValueError:
        return
    raise AssertionError("delivery_fee(-1) did not raise ValueError")


# Call every function whose name starts with test_, as a test runner does.
for name, test in list(globals().items()):
    if name.startswith("test_"):
        try:
            test()
        except AssertionError as error:
            print("FAIL", name, error)
        else:
            print("PASS", name)

PASS test_small_order_pays_the_fee
PASS test_fee_is_waived_from_499
PASS test_negative_total_is_refused

Each test calls the function with one input and asserts one fact about the result. assert condition does nothing when the condition is true and raises AssertionError when it is false; the loop at the bottom calls every function whose name starts with test_ and reports which ones raised. That loop is a tiny test runner. Python’s unittest module and the pytest package are full-size runners, and the next lessons of this module use them; the ideas in this lesson are the same in all three. Write tests while you write the code, run them before every change you keep, and add one for every bug you fix, so that it cannot come back unnoticed.

A test earns its keep on the day the code breaks. Here the same tests meet a version of the rule with one character wrong, > where the rule says “from 499”. Step through the run: the runner calls each test, the test calls the rule, and the failure travels back to the runner:

The program behind the figure prints the runner’s two lines. Press Edit, change > back to >= and run it again:

"""The same rule with a bug: an order of exactly ₹499 still pays the fee."""


def delivery_fee(order_rupees):
    if order_rupees < 0:
        raise ValueError(f"an order total cannot be negative: {order_rupees}")
    return 0 if order_rupees > 499 else 40  # the rule says "from ₹499", so this should be >=


def test_fee_is_waived_from_499():
    assert delivery_fee(499) == 0


def test_fee_is_waived_from_499_with_a_message():
    fee = delivery_fee(499)
    assert fee == 0, f"an order of ₹499 should travel free, but the fee was ₹{fee}"


for name, test in list(globals().items()):
    if name.startswith("test_"):
        try:
            test()
        except AssertionError as error:
            print("FAIL", name, repr(str(error)))
        else:
            print("PASS", name)

FAIL test_fee_is_waived_from_499 ''
FAIL test_fee_is_waived_from_499_with_a_message 'an order of ₹499 should travel free, but the fee was ₹40'

Both tests fail, but the first one says nothing about why: a bare assert raises an error with an empty message. The second adds a message after the comma, and the failure explains itself. pytest goes further and prints the values that were compared without any message, which is one reason many projects choose it.

Unit, integration and end-to-end tests differ in how much they run

Tests differ in how much of the program they run at once. The figure nests the three kinds, with the delivery rule as the example:

A unit test checks one function or class on its own, in memory, with nothing else taking part, so it runs in microseconds and points straight at the code that broke. An integration test checks your code working with a real part of the outside world, such as a database file, a folder of files or another module of the project. An end-to-end test runs the whole program the way its users do, through its command line or web page, and checks what they would see.

The speed difference is large enough to shape a whole test suite. This program checks the delivery rule 10,000 times in memory and 100 times through a real SQLite database file:

"""One rule checked by 10,000 unit tests and by 100 integration tests that use a real database file."""
import sqlite3
import tempfile
import time
from pathlib import Path


def delivery_fee(order_rupees):
    return 0 if order_rupees >= 499 else 40


def unit_test(total):
    # One function, in memory: nothing else takes part.
    assert delivery_fee(total) == (40 if total < 499 else 0)


def integration_test(db_file, total):
    # The order is saved to a database file and read back before the rule is checked.
    db = sqlite3.connect(db_file)
    with db:
        db.execute("CREATE TABLE IF NOT EXISTS orders (total INTEGER)")
        db.execute("INSERT INTO orders VALUES (?)", (total,))
    (saved,) = db.execute("SELECT total FROM orders ORDER BY rowid DESC LIMIT 1").fetchone()
    db.close()
    assert delivery_fee(saved) == (40 if total < 499 else 0)


def microseconds_each(test, totals):
    start = time.perf_counter()
    for total in totals:
        test(total)
    return (time.perf_counter() - start) * 1_000_000 / len(totals)


with tempfile.TemporaryDirectory() as folder:
    db_file = Path(folder) / "orders.db"
    unit = microseconds_each(unit_test, range(10_000))
    integration = microseconds_each(lambda total: integration_test(db_file, total), range(0, 1000, 10))

print(f"{'unit test, average:':27}{unit:12,.2f} µs")
print(f"{'integration test, average:':27}{integration:12,.2f} µs")
print(f"one integration test takes as long as about {integration / unit:,.0f} unit tests")

unit test, average:                0.10 µs
integration test, average:     2,766.47 µs
one integration test takes as long as about 28,572 unit tests

Writing to a disk and reading back costs thousands of times more than calling a function, so a suite of mostly unit tests finishes in a second and gets run constantly, while a suite of mostly slow tests gets run rarely and catches bugs late. Use integration tests for the places where parts meet, and keep end-to-end tests for the journeys that matter most.

A test that shares data with another fails for the wrong reason

A clear test has three steps: arrange the input and anything the code needs, act by calling the code once, then assert what must be true. Name the test after the behaviour it checks, such as test_fee_is_waived_from_499, so a failure report reads like a broken promise. And make each test fail for one reason only: a bug in the code it is about.

These two tests break that rule. The code they test is correct, yet the second test fails. Both add to the same list at the top of the module; step through the two calls and watch cart:

cart = []  # shared by both tests


def total(items):
    return sum(p for _, p in items)


def test_one_item():
    cart.append(("pen", 20))
    assert total(cart) == 20


def test_two_items():
    cart.append(("pen", 20))
    cart.append(("notebook", 60))
    assert total(cart) == 80, cart


test_one_item()
test_two_items()
print("both tests passed")

Error AssertionError: [('pen', 20), ('pen', 20), ('notebook', 60)]

The first test left its pen in the cart, so the second found three items and a total of 100 instead of 80. That is a failure for the wrong reason: the bug is in the tests, and it hides real failures behind false ones. The next program runs the same pair of tests in both orders, then rewrites them so that each arranges its own empty cart:

"""Two tests that share one cart, run in both orders; then the same tests with a cart each."""

cart = []  # one list for every test: the mistake


def add(cart, item, price):
    cart.append((item, price))


def total(cart):
    return sum(price for _, price in cart)


def test_one_item():
    add(cart, "pen", 20)
    assert total(cart) == 20


def test_two_items():
    add(cart, "pen", 20)
    add(cart, "notebook", 60)
    assert total(cart) == 80


def run(tests):
    for test in tests:
        try:
            test()
        except AssertionError:
            print(f"  FAIL {test.__name__}: the cart held {cart}")
        else:
            print(f"  PASS {test.__name__}")


print("one_item first:")
run([test_one_item, test_two_items])
cart.clear()
print("two_items first:")
run([test_two_items, test_one_item])


# Rewritten: each test arranges its own cart, so the order no longer matters.
def test_one_item_fresh():
    my_cart = []
    add(my_cart, "pen", 20)
    assert total(my_cart) == 20


def test_two_items_fresh():
    my_cart = []
    add(my_cart, "pen", 20)
    add(my_cart, "notebook", 60)
    assert total(my_cart) == 80


print("each test with its own cart, in both orders:")
run([test_one_item_fresh, test_two_items_fresh, test_two_items_fresh, test_one_item_fresh])

one_item first:
  PASS test_one_item
  FAIL test_two_items: the cart held [('pen', 20), ('pen', 20), ('notebook', 60)]
two_items first:
  PASS test_two_items
  FAIL test_one_item: the cart held [('pen', 20), ('notebook', 60), ('pen', 20)]
each test with its own cart, in both orders:
  PASS test_one_item_fresh
  PASS test_two_items_fresh
  PASS test_two_items_fresh
  PASS test_one_item_fresh

Whichever test runs second fails, and the rewritten tests pass in any order, as the last four lines show. Test runners help here: unittest gives each test a fresh object, and pytest gives each test fresh fixtures, but only if the tests do not share data of their own.

Bugs sit at the edges of a rule

Most bugs sit at the edges of a rule: the exact amount where a discount starts, an empty list, the largest value. Take a discount of 5 % from 1,000 rupees and 10 % from 5,000, capped at 750, written with two bugs: > at 5,000 and no cap. Four cases, two at each step of the rule, are enough to catch both. Step through the table of cases and watch got against expected:

def discount(total):  # has 2 bugs
    if total > 5000:
        return total * 10 // 100
    if total >= 1000:
        return total * 5 // 100
    return 0


cases = [  # (total, expected)
    (999, 0),
    (1000, 50),
    (5000, 500),
    (200000, 750),
]
failed = []
for total, expected in cases:
    got = discount(total)
    if got != expected:
        failed.append(total)
print("failed at", failed)

failed at [5000, 200000]

The cases at 999 and 1,000 passed; the case exactly on 5,000 and the case far above the cap failed, because of the four they are the only ones where the bugs change the answer. Before writing tests for a function, list the inputs that sit on or next to each edge: for each rule, the values just below, on and just above every boundary valueboundary value: A value at the edge of a case, such as 100 and 101 when the first slab of a bill ends at 100 units. Mistakes with < and <= show only there, so tests try each edge and the value just past it. More in “Decisions with if, elif and else”; zero and empty input; the largest value you expect; a value of the wrong type; and anything the rule refuses, such as a negative total. Here is that list as a table of nine cases, run against the correct and the buggy discount:

"""An edge-case table for a discount rule, run against a correct and a buggy version.

The rule: no discount below ₹1,000; 5 % from ₹1,000; 10 % from ₹5,000, but never more than ₹750.
Totals are whole rupees, and a discount is rounded down to whole rupees.
"""


def discount(total):
    if not isinstance(total, int) or isinstance(total, bool):
        raise TypeError(f"total must be whole rupees (an int), not {type(total).__name__}")
    if total < 0:
        raise ValueError(f"total cannot be negative: {total}")
    if total >= 5000:
        return min(total * 10 // 100, 750)
    if total >= 1000:
        return total * 5 // 100
    return 0


def buggy_discount(total):
    if total < 0:
        raise ValueError(f"total cannot be negative: {total}")
    if total > 5000:  # > where the rule says "from ₹5,000"
        return total * 10 // 100  # and no cap
    if total >= 1000:
        return total * 5 // 100
    return 0


CASES = [  # (why the case matters, total, expected discount or exception)
    ("empty cart", 0, 0),
    ("just below the first step", 999, 0),
    ("exactly on the first step", 1000, 50),
    ("just below the second step", 4999, 249),
    ("exactly on the second step", 5000, 500),
    ("where the cap starts", 7500, 750),
    ("far above the cap", 200000, 750),
    ("negative total", -1, ValueError),
    ("a string, not a number", "1500", TypeError),
]


def check(function):
    failed = 0
    for why, total, expected in CASES:
        try:
            got = function(total)
        except Exception as error:
            got = type(error)
        if got != expected:
            failed += 1
            print(f"  FAIL {why}: discount({total!r}) gave {getattr(got, '__name__', got)}, expected {getattr(expected, '__name__', expected)}")
    print(f"  {len(CASES) - failed} of {len(CASES)} cases pass")


print("discount:")
check(discount)
print("buggy_discount:")
check(buggy_discount)

discount:
  9 of 9 cases pass
buggy_discount:
  FAIL exactly on the second step: discount(5000) gave 250, expected 500
  FAIL far above the cap: discount(200000) gave 20000, expected 750
  7 of 9 cases pass

Only two of the nine cases notice the buggy version. The case at 7,500 passes for both versions, because 10 % of 7,500 happens to equal the cap: only a total far above the cap proves the cap is there.

The clock and chance become inputs a test can choose

A function that reads the clock, the random number generator or the environment by itself has hidden inputs: its result changes with the day the test runs. The usual fix is to pass those values in as parameters, so a test can choose them and the program supplies the real ones. Step through both versions of a library rule and see where each one gets today’s date:

The program runs the tests of the second version, then the real call:

"""A library rule that reads the clock, and the same rule with the day passed in."""
import random
from datetime import date, timedelta


def days_overdue_hidden_clock(due):
    # Hard to test: the answer changes every day, because the function reads the clock itself.
    return max(0, (date.today() - due).days)


def days_overdue(due, today):
    # Easy to test: the caller passes the day in, so a test can choose it.
    return max(0, (today - due).days)


def test_on_the_due_day():
    assert days_overdue(date(2026, 3, 1), today=date(2026, 3, 1)) == 0


def test_across_a_leap_day():
    # 2028 is a leap year: 28 February to 1 March is two days.
    assert days_overdue(date(2028, 2, 28), today=date(2028, 3, 1)) == 2


def test_returned_early():
    assert days_overdue(date(2026, 3, 10), today=date(2026, 3, 1)) == 0


def pick_winner(entries, rng):
    # Randomness is an input too: pass in the random number generator.
    return rng.choice(sorted(entries))


def test_draw_is_repeatable():
    entries = {"asha", "ravi", "meera", "john"}
    assert pick_winner(entries, random.Random(7)) == pick_winner(entries, random.Random(7))


for name, test in list(globals().items()):
    if name.startswith("test_"):
        test()
        print("PASS", name)

# The program itself passes the real day in, once, where it starts.
today = date.today()
print("a book due three days ago is", days_overdue(today - timedelta(days=3), today), "days overdue")

PASS test_on_the_due_day
PASS test_across_a_leap_day
PASS test_returned_early
PASS test_draw_is_repeatable
a book due three days ago is 3 days overdue

days_overdue_hidden_clock(due) can only be tested by computing the right answer from today’s date inside the test, which repeats the code’s own logic. days_overdue(due, today) takes the day as an argument, so each test fixes it: the due day itself, a leap day in between, a book returned early. Randomness works the same way: pick_winner() receives a random.Random object, and a test that gives it a seedseed: The starting value of a pseudo-random generator. The same seed gives the same stream of numbers, so random.seed(42) or random.Random(42) makes a simulation or a test repeatable. More in “Random numbers, simulations and statistics” of 7 gets the same winner every time. The program reads the real clock once, at its start, and passes the value down. Code shaped like this, with calculations in plain functions and the reading of clocks, files and input kept at the edges, is easier to test and to reuse.

Practise: tests that catch six bugs

The checks of this exercise run your tests against correct rules and against six broken versions. Every test must pass on the correct code, and every broken version must make at least one test fail.

from rules import discount, leap_year


def test_2024_is_a_leap_year():
    assert leap_year(2024) is True


# Write more tests: other kinds of year for leap_year, and the steps, the cap, the rounding and the
# negative total of discount.

"""The checks of the exercise: they run your tests (test_rules.py) against the correct rules and against six
versions of them that each contain one bug.

Every test of yours must pass on the correct rules, and each buggy version must make at least one of them fail. The
rules are defined here and given to your file as the module `rules`, so `from rules import discount, leap_year`
works in the browser. On your own computer, put the two correct functions in a file named rules.py.
"""

import importlib
import sys
import types


def leap_year(year):
    return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)


def discount(total):
    if total < 0:
        raise ValueError(f"a total cannot be negative: {total}")
    if total < 2000:
        return 0
    return min(total // 10, 500)


# ---- six versions with one bug each --------------------------------------------------------------------------------
def leap_every_fourth_year(year):
    return year % 4 == 0


def leap_without_the_400_rule(year):
    return year % 4 == 0 and year % 100 != 0


def discount_misses_2000(total):
    if total < 0:
        raise ValueError(f"a total cannot be negative: {total}")
    if total <= 2000:
        return 0
    return min(total // 10, 500)


def discount_without_the_cap(total):
    if total < 0:
        raise ValueError(f"a total cannot be negative: {total}")
    if total < 2000:
        return 0
    return total // 10


def discount_rounds_up(total):
    if total < 0:
        raise ValueError(f"a total cannot be negative: {total}")
    if total < 2000:
        return 0
    return min(-(-total // 10), 500)


def discount_accepts_negative(total):
    if total < 2000:
        return 0
    return min(total // 10, 500)


rules = types.ModuleType("rules")
sys.modules["rules"] = rules


def _your_module(leap, disc):
    """A fresh import of test_rules.py that sees these two functions as rules.leap_year and rules.discount."""
    rules.leap_year, rules.discount = leap, disc
    sys.modules.pop("test_rules", None)
    return importlib.import_module("test_rules")


def _your_tests(module):
    return [(name, value) for name, value in vars(module).items() if name.startswith("test") and callable(value) and getattr(value, "__module__", "") == module.__name__]


def _failing(leap, disc):
    """The names of your tests that fail (or raise anything) with these rules."""
    failed = []
    for name, test in _your_tests(_your_module(leap, disc)):
        try:
            test()
        except Exception:
            failed.append(name)
    return failed


def _caught(leap, disc):
    return not _failing(leap_year, discount) and bool(_failing(leap, disc))


def test_at_least_four_tests():
    """test_rules.py has at least four test functions"""
    count = len(_your_tests(_your_module(leap_year, discount)))
    if count < 4:
        raise AssertionError(f"test_rules.py has {count} test function(s): write at least four")


def test_all_pass_on_the_correct_rules():
    """every one of your tests passes on the correct rules"""
    failed = _failing(leap_year, discount)
    if failed:
        raise AssertionError(f"these fail although the rules are correct: {', '.join(failed)}")


def test_catches_every_fourth_year():
    """a test fails when every year divisible by 4 counts as a leap year"""
    if not _caught(leap_every_fourth_year, discount):
        raise AssertionError("no test fails when 1900 is called a leap year")


def test_catches_the_missing_400_rule():
    """a test fails when a century divisible by 400 is not a leap year"""
    if not _caught(leap_without_the_400_rule, discount):
        raise AssertionError("no test fails when 2000 is not a leap year")


def test_catches_the_step_at_2000():
    """a test fails when a total of exactly ₹2,000 gets no discount"""
    if not _caught(leap_year, discount_misses_2000):
        raise AssertionError("no test fails when ₹2,000 gets no discount")


def test_catches_the_missing_cap():
    """a test fails when the discount can exceed ₹500"""
    if not _caught(leap_year, discount_without_the_cap):
        raise AssertionError("no test fails when ₹9,000 gets ₹900 off")


def test_catches_rounding_up():
    """a test fails when the discount is rounded up instead of down"""
    if not _caught(leap_year, discount_rounds_up):
        raise AssertionError("no test fails when ₹2,005 gets ₹201 off instead of ₹200")


def test_catches_a_negative_total():
    """a test fails when a negative total is accepted"""
    if not _caught(leap_year, discount_accepts_negative):
        raise AssertionError("no test fails when discount(-1) returns 0 instead of raising ValueError")

The self-check below draws on all five ideas of the lesson and starts with whichever one your checkpoint answers show to be the weakest.

Interview questions

Fresher · What is the difference between a unit test and an integration test?

A unit test checks one small piece of code, usually one function or class, on its own and in memory: it calls the code with chosen inputs and asserts on the result, with no database, files or network involved. That makes it fast, microseconds to milliseconds, and when it fails it points straight at the code that broke. An integration test checks that pieces work together, such as your code with a real database, the file system or another module, so it can catch problems no unit test sees: a wrong query, a file format the two sides disagree on, a wrong argument order between modules. It is slower and a failure says less about where the fault is. A healthy suite has many unit tests and fewer integration tests, plus a few end-to-end tests of the whole program.

An interviewer usually follows a first answer with a case. Pick the answer you would give to each:

Key takeaways

• A test runs your code with an input you chose and asserts what the result must be, so a plain function with assert is already a test, and a message after the comma makes its failure explain itself.
• Keep assert for tests, because python -O removes it; program code raises real exceptions instead.
• Unit tests check one piece in memory and run in microseconds, integration tests add real parts such as a database file, and end-to-end tests run the whole program, so a suite has many of the first kind and few of the last.
• Each test arranges its own data, acts once and asserts, so that it fails only when the code it is about is wrong.
• Bugs sit at the edges of a rule, so test on and around every boundary, empty input, the largest value and the wrong type.
• Pass the clock and the random generator in as parameters, so that tests can choose them and the program reads the real ones once, at its start.