Python Module 21 – Python everywhere: data, web, AI and next steps
Jupyter notebooks: workflow and pitfalls
Use Jupyter notebooks without fooling yourself: hidden state from cells run out of order, IPython magics, rich display, clean diffs and tested modules.
What you will learn
- Read the JSON of an .ipynb file and say what saving and committing it keeps
- Detect hidden state by restarting the kernel and running every cell
- Use IPython magics such as %timeit, %run and %debug, and rich display
- Remove notebook outputs before a commit, and explain why it matters
- Move notebook code into modules with tests, and import them back
Before you start
What you’ll learn · Before you start
What you will learn
- Read the JSON of an .ipynb file and say what saving and committing it keeps
- Detect hidden state by restarting the kernel and running every cell
- Use IPython magics such as %timeit, %run and %debug, and rich display
- Remove notebook outputs before a commit, and explain why it matters
- Move notebook code into modules with tests, and import them back
Before you start
On this page
Run a notebook’s cells in the order that suits you, run one of them twice, save, and the file can show a total of 548.25 that its own code can no longer produce. A Jupyter notebook is a document of cells, code and text between them, whose code runs in a The separate process that runs a notebook's code, for Python usually IPython. It stays alive while the notebook is open and keeps every name a cell creates.: a separate Python process that keeps every name a cell creates for the cells that run after it.
That memory makes notebooks good for exploring data one step at a time, and it is also their main pitfall. Here you take apart a real notebook file, catch its hidden state by running it from a fresh start, rebuild what two IPython features do with the standard library, keep outputs out of version control, and move the code worth keeping into a tested module.
A notebook file is JSON that keeps outputs beside the code
A notebook is saved as an .ipynb file, which is JSON. The Jupyter documentation describes it as the inputs and outputs
of an interactive session, and the nbformat specification fixes its shape: a dictionary with metadata, the format
version (nbformat 4) and a list of cells. Every cell has a cell_type, its source and metadata; a code cell
also stores its outputs and an
The number Jupyter shows beside a code cell and saves with it: the position of the cell's last run among all the runs of the session.,
the number shown beside the cell, which records the order of its last run. Since nbformat 4.5 each cell also has an
id. Step through the file this lesson works with, analysis.ipynb, saved after an ordinary session:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- One JSON dictionary. analysis.ipynb is a text file of JSON. At the top level it holds three things: metadata, the format version and the list of cells.
- metadata. metadata names the kernel the notebook runs on, python3, and the Python version it was saved with, 3.14.8.
- The format version. nbformat is 4 and nbformat_minor 5. Since version 4.5 every cell has an id, such as load or add-item, that stays the same when outputs change.
- A markdown cell. cells lists the five cells in page order. The first is markdown: a cell_type, an id, its source and metadata, and nothing else.
- A code cell. A code cell also stores its outputs and an execution_count. Cell 1 saved the output 3 prices with the count 1: its last run was the first of the session.
- Counts out of order. The other code cells show the counts 6, 3 and 5. A count records the order of a cell's last run, so cells further down ran before the rate cell above them.
- The saved outputs. Cell 3 saved 5 prices and cell 4 saved total after discount: 548.25. Both are what those cells printed when they last ran, in the session that is now over.
- What is not in the file. The values of prices, rate or discount are nowhere in the file. They lived in the kernel. The file keeps the code of each cell and what the cells printed.
The source and the text of outputs may be stored as one string or as a list of strings, so code that reads notebooks has to handle both. Three consequences follow from the format. A notebook carries its outputs, so it can show results that the current code no longer produces. Every run changes the execution counts and outputs, so a notebook that you only re-ran shows up as changed in version control. And outputs can hold anything a cell printed, including personal data from a dataset or a token you printed to check it.
Key idea
A saved notebook keeps each cell’s code and what that cell printed when it last ran, never the values the kernel held, so its outputs can show results that its code no longer produces.
Checkpoint
Check: What an .ipynb file stores
The kernel keeps what earlier runs left behind
The editors and notebooks lesson of this track showed hidden state in three
cells. analysis.ipynb has it too: its prices were loaded, an item was appended by a cell that ran twice, a variable
called discount was renamed to rate and its cell run again, and the total was computed in between. Step through that
session, then the restart:
The drawing shows the whole scene; its 10 steps are listed below.
Every step as text
- Run 1: load. You run the load cell. The kernel makes prices, a list of three numbers, and the cell prints 3 prices.
- Run 2: add-item. You run the add-item cell. prices.append(150) gives the list a fourth item, and the cell prints 4 prices.
- Run 3: add-item again. You run the same cell again. It appends 150 a second time and prints 5 prices, the output that will be saved.
- Run 4: a discount. The cell that will later hold the rate says discount = 0.15 at this point, and you run it.
- Run 5: the total. The total cell adds the five prices, 645, and takes 15 % off: it prints total after discount: 548.25.
- Run 6: renamed. You rename discount to rate in that cell, change it to 0.1 and run it. The kernel now holds both names: rate, and the old discount, which no cell defines any more.
- Saved. You save. The file gets every cell's code as it is now, the outputs 3 prices, 5 prices and 548.25, and the counts 1, 6, 3 and 5. The names in the kernel are not saved.
- Restart and run all. Restarting ends the kernel process, and a new one starts with no names at all. Run all then runs the code cells from the top, in page order.
- 4 prices, not 5. load makes three prices and add-item appends once, so the cell prints 4 prices where the file shows 5 prices.
- NameError. The total cell reads discount, and no cell defines it: NameError: name 'discount' is not defined. The saved 548.25 came from a session that no one can repeat.
The program below does what “restart the kernel and run all” does: it runs the code cells of both notebooks from the top in one new namespace, and compares what they print with what was saved:
"""Restart and run all: run a notebook's code cells top to bottom in one new namespace, and compare with what was saved."""
import contextlib
import io
import json
from pathlib import Path
def text_of(value):
"""nbformat stores multi-line text as one string or as a list of strings."""
return "".join(value) if isinstance(value, list) else value
def saved_output(cell):
return "".join(text_of(o["text"]) for o in cell["outputs"] if o["output_type"] == "stream")
def run_all(path):
notebook = json.loads(Path(path).read_text(encoding="utf-8"))
cells = notebook["cells"]
order = [c["execution_count"] for c in cells if c["cell_type"] == "code"]
print(f"{path}: the saved run counts of the code cells are {order}")
namespace = {"__name__": "__main__"} # a fresh kernel knows nothing
for index, cell in enumerate(cells):
if cell["cell_type"] != "code":
continue
printed = io.StringIO()
try:
with contextlib.redirect_stdout(printed):
exec(compile(text_of(cell["source"]), f"<cell {index}>", "exec"), namespace)
except Exception as error:
print(f" cell {index}: {type(error).__name__}: {error}")
return False
now, before = printed.getvalue(), saved_output(cell)
verdict = "as saved" if now == before else f"saved output was {before!r}"
print(f" cell {index}: printed {now!r}, {verdict}")
return True
for name in ["analysis.ipynb", "fixed.ipynb"]:
print("reproducible:", run_all(name))Output
analysis.ipynb: the saved run counts of the code cells are [1, 6, 3, 5] cell 1: printed '3 prices\n', as saved cell 2: printed '', as saved cell 3: printed '4 prices\n', saved output was '5 prices\n' cell 4: NameError: name 'discount' is not defined reproducible: False fixed.ipynb: the saved run counts of the code cells are [1, 2, 3] cell 1: printed '4 prices\n', as saved cell 2: printed '', as saved cell 3: printed 'total after discount: 445.50\n', as saved reproducible: True
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 run_all.py
The run counts [1, 6, 3, 5] already gave it away: a cell lower down ran before a cell above it. Run from the top, the
notebook printed 4 prices where 5 prices was saved, because the append cell had run twice in the session, and then
failed with NameError, because no cell defines discount any more; the old kernel still had it. The fixed notebook
defines every name it uses, in order, and gives the same results however often it runs.
Warning
Before you trust a notebook’s numbers, share it or commit it, restart the kernel (in Jupyter, Kernel › Restart) and run every cell from the top. If any result changes or a cell fails, the notebook depended on hidden state.
Restarting and running every cell is the only check that covers everything. For long notebooks a static check helps
too: reading each cell’s abstract syntax tree: The tree of nodes that ast.parse() builds from source code: one node for each statement and expression, such as Assign, BinOp or Call, with their parts as child nodes. Building it runs none of the code. More in “The ast module: read code without running it” with the ast module, without running
anything, shows names that are read before any cell above defines them. You write exactly that checker in the exercise
of this lesson.
This site’s in-browser runner works the other way. It uses Pyodide, the CPython build for WebAssembly, but every Run starts a fresh interpreter, so nothing carries over between runs: no hidden state, and no step-by-step exploration either. For a notebook that runs entirely in the browser there is JupyterLite, a JupyterLab distribution without a server process; its Python kernels are Pyodide or Xeus Python, and its documentation notes that not every JupyterLab feature works there.
Running a cell that changes data a second time
prices.append(150) run twice appends twice, and nothing on the page shows it. Build the data in one cell that can
run any number of times, prices = [120, 95, 130, 150] as fixed.ipynb does, or restart before you run again.
Checkpoint
Check: Hidden state in the kernel
Magics are IPython commands, and %timeit times in batches
The kernel behind a Python notebook is IPython (the ipykernel package), and it adds
A command of IPython, not of Python, that starts with %. A line magic such as %timeit works on its own line; a cell magic such as %%timeit, on a cell's first line, applies to the whole cell.,
commands that start with % and are not Python. A line magic (%name) works on its own line; a cell magic
(%%name) on the first line of a cell applies to the whole cell. The ones worth knowing first:
%timeit stmttimes a statement;%%timeit setuptimes the rest of the cell, running the setup line once without timing it.%run script.pyruns a file as a program, in its own namespace, and then adds the names it defined to the notebook;%run -iruns it inside the notebook’s namespace instead.%debugopens the debugger on the exception that was just raised, so you can inspect the variables of every frame.%whoand%whoslist the variables you defined, which is a quick way to see hidden state.%load_ext autoreloadfollowed by%autoreload 2reloads imported modules before each cell runs, so edits to their files take effect at once.
%timeit uses the standard library’s timeit module. Without -n it picks a number of loops that takes long enough to
measure, runs seven batches (-r 7), and reports the mean and standard deviation per loop. This program does the same
with timeit alone, and prints the other summary, the best batch, which the module’s own command-line interface
reports:
"""What %timeit reports, rebuilt with the timeit module: loops picked automatically, 7 runs, mean and spread."""
import statistics
import timeit
timer = timeit.Timer("sum(range(1000))")
loops, _ = timer.autorange() # enough loops for the batch to take at least 0.2 s, as %timeit does without -n
runs = timer.repeat(repeat=7, number=loops) # %timeit's default of -r 7
per_loop = [seconds / loops for seconds in runs]
mean, spread = statistics.mean(per_loop), statistics.stdev(per_loop)
print(f"%timeit style: {mean * 1e6:.1f} µs ± {spread * 1e6:.2f} µs per loop "
f"(mean ± std. dev. of 7 runs, {loops:,} loops each)")
print(f"timeit style: {loops:,} loops, best of 7: {min(per_loop) * 1e6:.1f} µs per loop")Output
%timeit style: 22.3 µs ± 5.60 µs per loop (mean ± std. dev. of 7 runs, 20,000 loops each) timeit style: 20,000 loops, best of 7: 16.3 µs per loop
This output changes from run to run: timings depend on the computer and on what else it is doing, so every run prints different microseconds.
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 timeit_like_ipython.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through what the recorded run measured, from the loop count to the two summaries:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- One short statement. The statement is sum(range(1000)). One run takes microseconds, far too short to time well on its own, so it is timed in loops.
- Count up. Timer.autorange() times 1 loop, then 2, 5, 10, 20, 50 and so on, until one count takes at least 0.2 seconds. %timeit, given no -n, also picks a count that takes long enough to measure.
- 20,000 loops. In the recorded run that count was 20,000 loops, the first from the sequence that took at least 0.2 seconds.
- Seven runs. Then come seven runs of 20,000 loops each, %timeit's default of -r 7. Each run is timed as a whole.
- Time per loop. Each run's time divided by 20,000 is the time of one loop, so the seven runs give seven times per loop.
- The mean, as %timeit. %timeit reports the mean of the seven and their standard deviation. In the recorded run: 22.3 µs ± 5.60 µs per loop.
- The best, as timeit. The timeit module reports the fastest of the runs instead, 16.3 µs here. Its documentation advises that minimum: slower runs are usually other programs getting in the way.
The mean can never be below the best batch, and on a busy computer it is far above it, because some batches were
interrupted; the spread after ± shows how noisy the measurement was. The timeit documentation itself advises the
minimum, since slower runs are typically caused by other processes rather than by Python. For comparing two versions of
a piece of code, the best batch is the steadier number, as the timeit lesson
explains. Press Run to measure on your own computer: your numbers will differ from the recorded ones.
Checkpoint
Check: IPython magics and %timeit
A notebook shows the richest form an object offers
When the last expression of a cell is an object, the notebook asks it for its richest representation. Objects offer
them through methods with single underscores, _repr_html_(), _repr_markdown_(), _repr_png_() and others; the
IPython documentation says that without them the ordinary repr() is used, and that a method returning None counts as
missing. Your own classes can use them to appear as tables, formatted text or images:
"""Rich display: a notebook shows an object through its _repr_html_(); terminals and logs use its repr()."""
import html
class MarksTable:
def __init__(self, rows):
self.rows = rows
def __repr__(self):
return f"MarksTable({len(self.rows)} rows)"
def _repr_html_(self):
# Data can contain <, > and &: escape it, or a name becomes markup in the notebook.
body = "".join(f"<tr><td>{html.escape(name)}</td><td>{mark}</td></tr>" for name, mark in self.rows)
return f"<table><tr><th>Student</th><th>Mark</th></tr>{body}</table>"
table = MarksTable([("Asha", 78), ("<b>Ben</b>", 55)])
print("repr(): ", repr(table)) # what print(), a terminal or a log shows
print("_repr_html_(): ", table._repr_html_()) # what a notebook renders as a tableOutput
repr(): MarksTable(2 rows) _repr_html_(): <table><tr><th>Student</th><th>Mark</th></tr><tr><td>Asha</td><td>78</td></tr><tr><td><b>Ben</b></td><td>55</td></tr></table>
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 rich_display.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through the two ways the same table reaches the screen, displayed and printed:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Displayed, not printed. When the last line of a cell is an expression, here table, the notebook displays its value instead of printing it.
- Rich methods first. Displaying asks the object for its richer forms, methods such as _repr_html_(), _repr_markdown_() and _repr_png_(). MarksTable defines _repr_html_().
- An HTML table. _repr_html_() returns the HTML of a table, and the notebook renders it: a header row, then the rows of Asha and Ben.
- Data stays data. html.escape() turned <b>Ben</b> into <b>Ben</b>, so the name appears exactly as typed instead of in bold.
- None counts as missing. If _repr_html_() did not exist, or returned None, the notebook would treat it as missing and use the standard repr() instead.
- print() is plain text. print(table) never asks for HTML. It writes str(table), which falls back to __repr__: MarksTable(2 rows), as a terminal or a log would show it.
print() and logs still use repr(), so keep both. The HTML method escapes every value with html.escape(): a student
called <b>Ben</b> must appear as those characters, not as bold text, and data from files or users can contain far
worse than <b>. Press Edit, take the html.escape() call out of _repr_html_() and run it: the second row now
carries the name as markup. The method also has no side effects, as the IPython documentation asks, because the
notebook may call it whenever it displays the object.
Checkpoint
Check: Rich display of your own objects
Clearing outputs keeps diffs and secrets out of git
Clearing outputs before a commit keeps diffs to what really changed and keeps printed data out of the repository. The
command jupyter nbconvert --clear-output --inplace analysis.ipynb does it for one file. This program does the same to
analysis.ipynb and shows what that removes from one cell:
"""Clear a notebook's outputs and run counts, as `jupyter nbconvert --clear-output` does, and show what changes."""
import copy
import json
from pathlib import Path
def clear_outputs(notebook):
clean = copy.deepcopy(notebook)
for cell in clean["cells"]:
if cell["cell_type"] == "code":
cell["outputs"] = []
cell["execution_count"] = None
return clean
def dump(notebook):
# Jupyter writes notebooks with sorted keys and an indent of one space.
return json.dumps(notebook, indent=1, sort_keys=True, ensure_ascii=False) + "\n"
saved = json.loads(Path("analysis.ipynb").read_text(encoding="utf-8"))
clean = clear_outputs(saved)
before, after = dump(saved).splitlines(), dump(clean).splitlines()
print(f"lines before: {len(before)}, after: {len(after)}")
print("the add-item cell as saved:")
print(json.dumps(saved["cells"][3], indent=1, sort_keys=True))
print("the same cell, cleared:")
print(json.dumps(clean["cells"][3], indent=1, sort_keys=True))
print("code unchanged:", [c["source"] for c in saved["cells"]] == [c["source"] for c in clean["cells"]])Output
lines before: 93, after: 69
the add-item cell as saved:
{
"cell_type": "code",
"execution_count": 3,
"id": "add-item",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5 prices\n"
]
}
],
"source": [
"prices.append(150)\n",
"print(len(prices), \"prices\")"
]
}
the same cell, cleared:
{
"cell_type": "code",
"execution_count": null,
"id": "add-item",
"metadata": {},
"outputs": [],
"source": [
"prices.append(150)\n",
"print(len(prices), \"prices\")"
]
}
code unchanged: True
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 clear_outputs.py
The file shrank from 93 lines to 69, and only the outputs and the execution_count changed; the source and the cell id
are the same. Teams usually automate the step with a
A script that git runs before it records a commit, which can check or change the staged files, for example by clearing every notebook's outputs.,
so that nobody has to remember it. The notebook with outputs can still be shared as an export, an HTML or PDF file of a
run you made from a restarted kernel.
Printing a secret to check it
print(api_key) in a cell saves the key in the notebook’s outputs, and a commit publishes it. Check that it is set
with print(bool(api_key)) instead, and clear the outputs before every commit.
Checkpoint
Check: Notebooks without outputs in git
Code worth keeping moves into a module with tests
A notebook is a good place to work out what the code should do and a poor place to keep it: cells cannot be imported by
other programs, and they are hard to test. Once a function works, move it into a .py file next to the notebook, give
it a docstring with an example or two, and import it in the notebook. With %autoreload 2 active, edits to the module
take effect in the next cell you run, without restarting the kernel.
extracted/check_cleaning.py
"""Check cleaning.py the way a test run would: its doctests, then edge cases the notebook never met."""
import doctest
import cleaning
results = doctest.testmod(cleaning)
print(f"doctests: {results.attempted} run, {results.failed} failed")
assert cleaning.parse_price("₹0.5") == 50, "half a rupee is 50 paise"
assert cleaning.parse_price(".75") == 75, "no rupees at all"
assert cleaning.clean_rows([]) == []
print("edge cases: passed")Output
doctests: 3 run, 0 failed edge cases: passed
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_cleaning.py
extracted/cleaning.py
"""Cleaning helpers moved out of a notebook into a module, so that notebooks import them and tests check them."""
def parse_price(text):
"""Turn a price typed by a person into paise.
>>> parse_price("₹1,299.50")
129950
>>> parse_price(" 99 ")
9900
"""
cleaned = text.strip().replace("₹", "").replace(",", "")
rupees, _, paise = cleaned.partition(".")
return int(rupees or "0") * 100 + int((paise + "00")[:2])
def clean_rows(rows):
"""Keep the rows that have a name and a price, as (name, paise) pairs.
>>> clean_rows([{"name": "Pen", "price": "10"}, {"name": "", "price": "5"}])
[('Pen', 1000)]
"""
kept = []
for row in rows:
name, price = row.get("name", "").strip(), row.get("price", "").strip()
if name and price:
kept.append((name, parse_price(price)))
return keptRuns on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
cleaning.py holds two helpers that started as notebook cells. Their docstrings contain examples that
doctest: The standard library module that finds examples written after >>> in docstrings, runs them and reports each one whose output differs from the line below it. More in “Docstrings, comments and PEP 8 style” runs, so the documentation is checked too, and check_cleaning.py adds edge cases that
the notebook’s data never contained, such as a price without rupees. Press Edit, add a check of your own to
check_cleaning.py, such as assert cleaning.parse_price("₹5") == 500, and run it. In a project the checks would live
in a test file run by pytest or unittest, and the notebook would shrink to loading data, calling tested functions and
showing results. The table compares the three places code can live:
| Criterion | Notebook (.ipynb) | Script (.py) | Module with tests |
|---|---|---|---|
| Best for | Exploring data and explaining results | Jobs that run from start to end | Code other code uses |
| State between runs | The kernel keeps everything | None: every run starts fresh | None: tests start fresh |
| Diffs in version control | Noisy unless outputs are cleared | Clean | Clean |
| Testing | Hard: restart and run all | Run it, or test its functions | doctest, unittest or pytest |
| When to choose | Asking questions of data | Repeatable tasks | Anything you want to keep |
The whole workflow is one loop: explore, prove that the notebook runs from a fresh kernel, then keep what works in a tested module and commit without outputs:
A notebook workflow that stays reproducible
Text description of the diagram
The diagram is a flow from top to bottom, with one loop.
- Explore in cells: load the data, look at it and try ideas.
- Restart the kernel and run every cell from the top.
- The decision: are the results the same as before?
- If not, fix the hidden state, by reordering cells or defining the missing names, and restart and run all again. This loop repeats until the results match.
- If they are the same, move the code you want to keep into a .py module with tests.
- Import that module in the notebook, with the autoreload extension so that edits to the module are picked up.
- Clear the outputs, then commit the notebook and the module.
Checkpoint
Check: From notebook cells to a tested module
Practise: audit a notebook for hidden state
The exercise asks for audit_notebook(), which reads a notebook’s JSON and reports the cells that never ran, the cells
that ran out of order and the names read before any cell defines them, all without running a line of it. Its sample
tests run in your browser:
Exercise · Medium · Python
Audit a notebook for hidden state
Write audit_notebook(notebook) in audit.py. notebook is a notebook as json.load() reads an .ipynb file: a dictionary whose "cells" list holds cells with a "cell_type" ("code" or "markdown") and a "source", which is a string or a list of strings to join; code cells also have an "execution_count", an integer or None. Positions below are indexes in the "cells" list, counting every cell. Return a dictionary with three keys:
"not_run": the positions of code cells whoseexecution_countisNone, in order."out_of_order": the positions of code cells that ran but have a lowerexecution_countthan some code cell above them that ran, in order."undefined":(position, name)pairs for each name a code cell reads that is not a built-in (dir(builtins)) and that no code cell above it and not the cell itself defines; sorted by position, then name, each pair once.
A cell defines every name it binds anywhere in its code: assignment targets (x = …, x += …, x: int = …, for x in …, with … as x, except … as x, (x := …)), def and class names, function parameters, and imported names (import a.b defines a, from m import x as y defines y). Lines whose first non-space character is % or ! are IPython magics or shell commands: drop them before you parse the cell with ast.parse().
Starter code · audit.py
import ast
import builtins
def audit_notebook(notebook):
"""Report cells that never ran, cells that ran out of order, and names read before any cell defines them."""
# Replace these lines with your code.
return {"not_run": [], "out_of_order": [], "undefined": []}The sample tests · test_audit.py
from audit import audit_notebook
def code(source, count):
return {"cell_type": "code", "execution_count": count, "metadata": {}, "outputs": [], "source": source}
def markdown(source):
return {"cell_type": "markdown", "metadata": {}, "source": source}
def notebook(*cells):
return {"cells": list(cells), "metadata": {}, "nbformat": 4, "nbformat_minor": 5}
def test_clean_notebook():
"""a notebook run top to bottom has nothing to report"""
nb = notebook(markdown("# Prices"), code("prices = [120, 95]", 1), code("print(sum(prices))", 2))
assert audit_notebook(nb) == {"not_run": [], "out_of_order": [], "undefined": []}
def test_not_run_and_out_of_order():
"""positions count every cell; a lower count below a higher one is out of order"""
nb = notebook(code("a = 1", 1), markdown("text"), code("b = 2", None), code("c = 3", 5), code("d = 4", 2), code("e = 5", 6))
report = audit_notebook(nb)
assert report["not_run"] == [2]
assert report["out_of_order"] == [4]
def test_names_read_before_they_are_defined():
"""a name defined only by a later cell, or by no cell, is undefined where it is read"""
nb = notebook(code("total = price * quantity", 1), code("price = 10", 2), code("print(price + tax)", 3))
assert audit_notebook(nb)["undefined"] == [(0, "price"), (0, "quantity"), (2, "tax")]
def test_every_kind_of_definition_counts():
"""imports, def, class, parameters, loops, with, except and := all define names"""
first = "\n".join([
"import os.path",
"from math import sqrt as root",
"def scale(values, factor=2):",
" return [v * factor for v in values]",
"class Report:",
" pass",
"for item in range(3):",
" count = item",
"with open(os.devnull) as handle:",
" pass",
"try:",
" pass",
"except ValueError as problem:",
" pass",
"if (n := 4) > 3:",
" pass",
])
second = "print(scale([1], 3), root(n), Report, count, handle, problem, os.sep, len([]))"
nb = notebook(code(first, 1), code(second, 2))
assert audit_notebook(nb)["undefined"] == []
def test_list_sources_and_magics():
"""sources may be lists of strings, and magic lines are not Python"""
nb = notebook(
code(["%load_ext autoreload\n", "data = [3, 1, 2]\n", "!ls\n"], 1),
code(["%timeit sorted(data)\n", "result = sorted(data)\n", "print(result, missing)"], 2),
)
assert audit_notebook(nb) == {"not_run": [], "out_of_order": [], "undefined": [(1, "missing")]}
def test_each_pair_once():
"""a name read twice in one cell is reported once for that cell"""
nb = notebook(code("print(x)\nprint(x + y)", 1), code("print(x)", 2))
assert audit_notebook(nb)["undefined"] == [(0, "x"), (0, "y"), (1, "x")]A hint
For each code cell, join its source, blank out the magic lines, and call ast.parse(). Walk the tree with ast.walk(): an ast.Name whose ctx is ast.Load is a read, any other ast.Name is a binding; add the names of ast.FunctionDef, ast.AsyncFunctionDef and ast.ClassDef nodes, every ast.arg, every named ast.ExceptHandler, and the names of ast.Import and ast.ImportFrom aliases. Keep one set of names defined by the cells you have passed, and compare each cell's reads with that set, its own bindings and the built-ins.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
When it passes, revisit the six ideas of this lesson, from the notebook’s JSON to tested modules. Questions on your weakest idea come first, and each pick is explained on the spot, wrong ones included:
Practice that adapts to you
Practise this lesson
17 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
What is stored in an .ipynb file?
An .ipynb file is JSON in the nbformat structure: notebook metadata (such as the kernel and language), the format
version, and a list of cells. Each cell has a type (code or markdown), its source and metadata, and, since nbformat
4.5, an id. Code cells also store their outputs (printed text, tables as HTML, images as base64 and error tracebacks)
and an execution count, the number of the run that last executed them. Because outputs and counts are saved, the file
can show results the current code no longer produces, every re-run changes the file, and anything printed (personal
data, tokens) ends up in it. That is why teams restart and run all before sharing and clear outputs before committing.
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
A teammate's notebook shows a saved model accuracy of 0.94, but when you run it from top to bottom you get 0.81. How do you work out what happened, and what would you change in the team's way of working?
-
Mid to senior
Your team keeps notebooks in git. Pull requests show hundreds of changed lines even when nobody touched the code, and once a customer's email address appeared in a committed output. What do you change, and in what order?
Key takeaways
- A notebook is JSON: cells with their source, and code cells with their outputs and execution counts, but never the values the kernel held.
- The kernel keeps every name between cells, so the page can show results that only an earlier session produced: restart the kernel and run all before you trust, share or commit a notebook.
%timeit,%run,%debug,%whosand%autoreloadcover most exploration;%timeittimes seven batches and reports their mean, while the best batch is the steadier figure for comparing code.- A notebook shows the richest form an object offers, such as
_repr_html_(), and falls back torepr(); HTML built from data must escape it. - Clearing outputs and run counts before each commit, with a pre-commit hook, keeps diffs to real changes and printed data out of the repository.
- Code you keep belongs in a module with docstring examples and tests, imported by the notebook.
References
- The Jupyter Notebook: notebook documents and kernels (Project Jupyter)
- The notebook file format (nbformat) (Project Jupyter)
- Kernels (programming languages) (Project Jupyter)
- Built-in magic commands (The IPython Development Team)
- Rich display of your own objects (The IPython Development Team)
- The autoreload extension (The IPython Development Team)
- nbconvert configuration options: --clear-output (Project Jupyter)
- JupyterLite documentation (Project Jupyter)
- timeit: measure execution time of small code snippets (Python Software Foundation)
- doctest: test interactive Python examples (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress