Python Module 21 – Python everywhere: data, web, AI and next steps
NumPy first steps: arrays and vectorisation
Create and index NumPy arrays, replace Python loops with vectorised operations and broadcasting, and catch the integer overflow that Python ints never have.
What you will learn
- Create NumPy arrays and index them by row, column, slice and mask
- Replace Python loops with vectorised operations and broadcasting
- Predict which edits of a slice or a selection reach the original array
- Predict when fixed-size integer types overflow, and prevent it
- Decide when to move on to pandas and the Data Analysis with Python track
Before you start
What you’ll learn · Before you start
What you will learn
- Create NumPy arrays and index them by row, column, slice and mask
- Replace Python loops with vectorised operations and broadcasting
- Predict which edits of a slice or a selection reach the original array
- Predict when fixed-size integer types overflow, and prevent it
- Decide when to move on to pandas and the Data Analysis with Python track
Before you start
On this page
A Python loop that totals a million order lines runs its body a million times; one NumPy expression gets the same exact total more than ten times faster on the computer that recorded this lesson’s runs. NumPy is the package most of scientific and data Python is built on, and its centre is one type, the NumPy's array type: a grid of values that all share one data type, stored side by side in memory, with a size fixed when it is made.: a grid of numbers of one type that NumPy works on in compiled code instead of in Python.
You will read and reduce a small table of marks, measure when that speed-up appears and when it does not, step through the rule that lets arrays of different shapes work together, see which edits of a slice reach the original array, and catch the integer overflow that Python’s own integers never have. The last section says when a table belongs in pandas instead.
An array is a grid of one dtype, read one axis at a time
Five students sat four subjects, so a class has 20 marks. np.array() turns the nested lists into one array with five
rows and four columns, and the program then reads it in five ways:
"""Marks of five students in four subjects, held in one NumPy array and read by row, column, element, slice and mask."""
import numpy as np
subjects = ["maths", "physics", "chemistry", "english"]
students = ["Asha", "Ben", "Chen", "Divya", "Emre"]
marks = np.array(
[
[78, 64, 81, 70],
[55, 72, 49, 88],
[91, 85, 77, 62],
[66, 58, 93, 75],
[42, 90, 68, 81],
]
)
print("shape:", marks.shape, "| ndim:", marks.ndim, "| size:", marks.size, "| dtype:", marks.dtype)
print("Chen, a row: ", marks[2])
print("physics, a column: ", marks[:, 1])
print("Divya in chemistry: ", marks[3, 2])
print("first two students, last two subjects:")
print(marks[:2, 2:])
below_50 = marks < 50 # a boolean array of the same shape
print("marks < 50, a boolean array of shape", below_50.shape)
print(below_50)
print("marks[marks < 50]:", marks[below_50])Output
shape: (5, 4) | ndim: 2 | size: 20 | dtype: int64 Chen, a row: [91 85 77 62] physics, a column: [64 72 85 58 90] Divya in chemistry: 93 first two students, last two subjects: [[81 70] [49 88]] marks < 50, a boolean array of shape (5, 4) [[False False False False] [False False True False] [False False False False] [False False False False] [ True False False False]] marks[marks < 50]: [49 42]
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 marks_table.py
Version note
The examples ran with NumPy 2.5.3 on CPython 3.14.8 and show their recorded output. NumPy is a third-party package,
and the in-browser runner of this site offers the standard library only, so the NumPy examples have no Run button.
To run them yourself, install NumPy in a virtual environment: python -m pip install numpy==2.5.3. The program in
this lesson that needs only the standard library, and the exercise at the end, run in your browser.
Four attributes describe every array. shape is the size along each dimension, (5, 4) here. ndim is the number of
dimensions, and NumPy calls each one an
One dimension of a NumPy array, counted from 0. In a table of shape (5, 4), axis 0 runs down the 5 rows and axis 1 across the 4 columns.:
axis 0 runs down the rows and axis 1 across the columns. size is the number of elements, 20, and dtype is the one
type they all share, the array’s
The one data type that every value of a NumPy array has, such as int64, int32 or float64. It fixes how many bytes each value takes and which values fit..
NumPy chose int64, its default integer on this 64-bit computer, because every value was a Python int. Unlike a
list, an array holds values of that one dtype only, and its size is fixed when it is created.
Indexing takes one position or slice per axis, separated by commas. Step through the five expressions the program printed, and watch which rows each one reads and what shape comes back:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The whole array. marks holds 20 marks in one array. Axis 0 runs down the five students, axis 1 across maths, physics, chemistry and english, so its shape is (5, 4), and every value is an int64.
- marks[2]: a row. One position picks along axis 0, the rows: marks[2] is the third row, Chen's marks. One axis is left, the four subjects, so the shape is (4,).
- marks[:, 1]: a column. The colon means every row, and 1 picks the second position on axis 1, physics. One value comes from each student, top to bottom: shape (5,).
- marks[3, 2]: one value. One position on each axis picks a single mark: row 3, Divya, and position 2 on axis 1, chemistry. No axis is left, so the result is one number, 93.
- marks[:2, 2:]: a block. A slice on each axis keeps both axes: rows 0 and 1, Asha and Ben, and positions 2 and 3, chemistry and english. The result is a smaller table of shape (2, 2).
- marks[marks < 50]: a mask. marks < 50 is a boolean array of shape (5, 4), True only at Ben's 49 in chemistry and Emre's 42 in maths. Used as an index, it picks those marks, in row order.
A single position removes its axis from the result: marks[2] is a row of shape (4,), and marks[3, 2], with a
position on both axes, is one number. A slice keeps its axis, so marks[:2, 2:] is still a table. A comparison such
as marks < 50 gives a
An array of True and False values, such as marks < 50, with the shape of the array it was made from. Used as an index, it selects the values where it is True. of the same shape, and using it as an
index selects the marks where it is True, as a new array.
Checkpoint
Check: Shape, dtype and indexing by axis
A reduction removes the axis you name
The mean of each subject and the mean of each student come from the same method; only its axis argument differs.
Methods such as sum(), mean(), max() and argmax() reduce an array: without an axis they use every value, and
with one they work along that axis only:
"""The marks table reduced: over the whole array, along axis 0 (one result per subject) and along axis 1 (per student)."""
import numpy as np
subjects = ["maths", "physics", "chemistry", "english"]
students = ["Asha", "Ben", "Chen", "Divya", "Emre"]
marks = np.array(
[
[78, 64, 81, 70],
[55, 72, 49, 88],
[91, 85, 77, 62],
[66, 58, 93, 75],
[42, 90, 68, 81],
]
)
print("sum of all 20 marks:", marks.sum(), "| their mean:", marks.mean())
print("mean per subject:", marks.mean(axis=0), "shape", marks.mean(axis=0).shape)
print("mean per student:", marks.mean(axis=1), "shape", marks.mean(axis=1).shape)
best = marks.argmax(axis=1) # the position of the largest mark in each row
print("position of each student's best mark:", best)
print("best subject of each student:", [subjects[i] for i in best])
below_50 = marks < 50
print("how many marks are below 50:", below_50.sum()) # True counts as 1
print("any mark below 50, per student:", below_50.any(axis=1))
print("students with a mark below 50:", [students[i] for i in np.flatnonzero(below_50.any(axis=1))])Output
sum of all 20 marks: 1445 | their mean: 72.25 mean per subject: [66.4 73.8 73.6 75.2] shape (4,) mean per student: [73.25 66. 78.75 73. 70.25] shape (5,) position of each student's best mark: [2 3 0 2 1] best subject of each student: ['chemistry', 'english', 'maths', 'chemistry', 'physics'] how many marks are below 50: 2 any mark below 50, per student: [False True False False True] students with a mark below 50: ['Ben', 'Emre']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 marks_reduce.py
axis=0 collapses the rows and gives one result per column, the mean of each subject; axis=1 collapses the columns
and gives one result per row, the mean of each student. The printed shapes show the rule: the axis you pass is the one
that disappears, so (5, 4) reduced along axis 0 has shape (4,). argmax(axis=1) follows the same rule but
returns positions, which the program turns into subject names. And because True counts as 1, the sum() of a mask
counts its True values, while any(axis=1) asks the question once per student. Step through the reductions and
watch the shape on the right lose the axis each call names:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- No axis named. Without an axis, a reduction uses every value of the array: marks.mean() adds all 20 marks, 1,445, and divides by 20. One number comes back, 72.25.
- axis=0 removes the rows. mean(axis=0) collapses axis 0: for each subject, the five marks down the rows become one mean. Axis 0 disappears from the shape, so (5, 4) becomes (4,).
- axis=1 removes the columns. mean(axis=1) collapses axis 1: each student's four marks become one mean, shown beside each row. Axis 1 disappears, so the shape is (5,).
- argmax names a position. argmax(axis=1) gives, for each row, the position of its largest mark: 2, 3, 0, 2 and 1. The program turns those positions into subject names.
- Counting with a mask. True counts as 1, so (marks < 50).sum() counts the marks below 50: 2. With axis=1, any() asks the question once per student, and only the rows of Ben and Emre answer True.
Key idea
The axis you pass to a reduction is the axis that disappears: a (5, 4) table gives (4,) along axis 0, (5,)
along axis 1, and a single value when no axis is named.
Checkpoint
Check: Reducing along an axis
One expression runs the whole loop in compiled code
The next program computes the value of a million order lines twice, once with a Python loop over plain integers and
once with the single expression (price_paise * quantity).sum():
"""One order total computed twice: with a Python loop and with one NumPy expression, each timed."""
import time
import numpy as np
rng = np.random.default_rng(seed=7)
n = 1_000_000
price_paise = rng.integers(5_000, 50_000, size=n) # whole paise, so both totals are exact integers
quantity = rng.integers(1, 10, size=n)
def total_with_loop(prices, quantities):
total = 0
for p, q in zip(prices, quantities):
total += p * q
return total
prices_list, quantities_list = price_paise.tolist(), quantity.tolist() # plain Python ints for the loop
start = time.perf_counter()
loop_total = total_with_loop(prices_list, quantities_list)
loop_seconds = time.perf_counter() - start
start = time.perf_counter()
vector_total = int((price_paise * quantity).sum())
vector_seconds = time.perf_counter() - start
print(f"{n:,} order lines")
print(f"loop total: {loop_total:,} paise")
print(f"vectorised total: {vector_total:,} paise")
print("same answer:", loop_total == vector_total)
print(f"loop {loop_seconds * 1000:.0f} ms, vectorised {vector_seconds * 1000:.1f} ms: "
f"about {loop_seconds / vector_seconds:.0f} times faster on this computer")Output
1,000,000 order lines loop total: 137,507,813,345 paise vectorised total: 137,507,813,345 paise same answer: True loop 30 ms, vectorised 1.4 ms: about 22 times faster on this computer
This output changes from run to run: the times depend on the computer and on what else it is doing, so every run prints different milliseconds; the factor always has at least two digits.
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 loop_or_vector.py
Both totals are exact, because the prices are whole paise and both versions add integers, so the answers agree to the last digit. On the computer that recorded this run the expression was more than ten times faster, and the last line of the output gives the exact factor. Writing one operation on whole arrays instead of a loop over their elements is called Writing one operation on whole arrays, such as price * quantity, instead of a Python loop over their elements, so that NumPy runs the loop in compiled code.. The loop has not gone away: NumPy’s documentation describes it as running “behind the scenes” in pre-compiled C code. Step through what each version does for its million lines:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- The same total twice. Both versions add up 1,000,000 order lines and get the same exact total, 137,507,813,345 paise. The left column is one pass of the loop over two lists of Python ints; the right column is everything NumPy does.
- Two objects per pass. A list holds references to objects, so each pass takes two int objects, a price and a quantity, and names them p and q.
- The type decides. An object's type decides what its operations do, so p * q is worked out from the types of p and q: int's own multiplication, done on these two objects.
- New objects every pass. Arithmetic returns new number objects: the product is a new int, and total += p * q makes another one for the running total.
- And again. Then the loop goes back for the next pair. Every step of the left column runs once per order line, 1,000,000 times in all.
- One call. price_paise * quantity is a single call. Every value of an array has the array's one dtype, here int64, so NumPy picks its compiled inner loop for int64 times int64 once, for the whole array.
- Raw values, side by side. The compiled loop runs through the two arrays, whose values lie side by side in memory, all the same size, and multiplies 1,000,000 pairs. No Python object is made for any of them.
- One array, one sum. The products fill one new int64 array, and .sum() adds them in a second compiled loop. int() turns that one result into a Python int for printing: 137,507,813,345.
The loop repeats its whole column for every order line, making new Python objects on every pass, while NumPy reads the dtype once and runs one compiled loop over raw values. That also explains the conditions of the speed-up. This program doubles numbers three ways, for 5 values and for a million:
"""Doubling numbers three ways, for 5 values and for 1,000,000: a list comprehension, NumPy from a list and back, and
NumPy on numbers already in an array. Each is timed with timeit, as the fastest of 5 repeats."""
import timeit
import numpy as np
def per_call(work, loops):
"""Seconds per call: the fastest of 5 repeats of `loops` calls, the steadiest figure on a busy computer."""
return min(timeit.repeat(work, number=loops, repeat=5)) / loops
times = {}
for n, loops in [(5, 20_000), (1_000_000, 3)]:
values = list(range(n)) # Python ints in a list, as data often arrives
ready = np.array(values) # the same numbers, already in an array
times[n] = (
per_call(lambda: [v * 2 for v in values], loops),
per_call(lambda: (np.array(values) * 2).tolist(), loops), # list in, list out
per_call(lambda: ready * 2, loops),
)
print(f"{n:,} values, time per call:")
for name, seconds in zip(["list comprehension", "NumPy from a list and back", "NumPy on a ready array"], times[n]):
print(f" {name:<27}{seconds * 1e6:>12,.2f} µs")
small, large = times[5], times[1_000_000]
print("5 values: the comprehension was the fastest of the three:", small[0] < min(small[1], small[2]))
print("1,000,000 values: the conversions cost more than the comprehension:", large[1] > large[0])
print("1,000,000 values: the ready array was over 10 times faster than the comprehension:", large[2] * 10 < large[0])Output
5 values, time per call: list comprehension 0.08 µs NumPy from a list and back 0.65 µs NumPy on a ready array 0.31 µs 1,000,000 values, time per call: list comprehension 22,153.65 µs NumPy from a list and back 35,794.90 µs NumPy on a ready array 242.11 µs 5 values: the comprehension was the fastest of the three: True 1,000,000 values: the conversions cost more than the comprehension: True 1,000,000 values: the ready array was over 10 times faster than the comprehension: True
This output changes from run to run: the times depend on the computer and on what else it is doing, so every run prints different microseconds; the three checks at the end hold in every run.
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 small_or_large.py
For 5 values the comprehension was the fastest of the three, even against NumPy on a ready array: with so few values, the work NumPy saves is smaller than what each call costs. For a million values, turning the list into an array and back took longer than the whole comprehension, and only on numbers already in an array was NumPy more than ten times faster. Keep data in arrays from the moment it is loaded, and measure speed-ups of your own with the methods of the timeit lesson, reporting the computer that produced them.
np.vectorize() is not vectorisation
np.vectorize(lambda p, q: p * q)(price_paise, quantity) looks vectorised, but NumPy’s documentation says the
function is provided for convenience, not for performance, and that it is essentially a for loop: the lambda still
runs once per pair. Write the operation on the whole arrays instead: price_paise * quantity.
Checkpoint
Check: Vectorised operations and their conditions
Broadcasting lines shapes up from the right
Arithmetic between an array and a single number already works: marks * 2 doubles every mark.
NumPy's rule for arithmetic on arrays of different shapes: compare the shapes from the right; two sizes fit when they are equal or one is 1, and a size of 1 stretches. extends that to two arrays
whose shapes differ. NumPy’s documentation states the rule:
- the shapes are compared from the right, one dimension at a time;
- two sizes are compatible when they are equal or one of them is 1;
- a missing dimension on the left counts as 1, so
(4,)meets(5, 4)as if it were(1, 4); - each size of 1 is stretched to the other size, and the result takes the larger size on each axis.
If any pair is incompatible, NumPy raises ValueError: operands could not be broadcast together. The stretching copies
nothing: the smaller array’s values are reused for each row or column. The figure puts the rule in order for (5, 4)
and (4,), the first pair of shapes in the program below:
How NumPy finds the shape of a result
Text description of the diagram
The diagram is a flow from top to bottom.
- Two shapes arrive, for example (5, 4) and (4,).
- They are lined up from the right, and the shorter shape is padded on the left with sizes of 1, so (4,) becomes (1, 4).
- The decision: is every pair of sizes either equal, or does it contain a 1?
- If yes, each axis of the result takes the larger of its two sizes, here (5, 4), and the size-1 axes are stretched without copying data.
- If any pair fails, NumPy raises ValueError: operands could not be broadcast together.
The program below applies the rule four times: a row against the table, a column against a row, a bonus that does not fit, and the same bonus turned into a column:
"""Broadcasting: a (5, 4) table against a (4,) row, a (3, 1) column against a (4,) row, and shapes that do not fit."""
import numpy as np
marks = np.array(
[
[78, 64, 81, 70],
[55, 72, 49, 88],
[91, 85, 77, 62],
[66, 58, 93, 75],
[42, 90, 68, 81],
]
)
# Each mark as a percentage of the top mark in its subject: (5, 4) / (4,) -> (5, 4).
top = marks.max(axis=0)
print("top mark per subject:", top, top.shape)
print((marks / top * 100).round(1))
# A column and a row make a whole table: (3, 1) + (4,) -> (3, 4).
column = np.array([[0], [10], [20]])
row = np.array([1, 2, 3, 4])
table = column + row
print("(3, 1) + (4,) gives", table.shape)
print(table)
# A bonus per student is a column, (5,) -> (5, 1); added as a (5,) row it fails.
bonus = np.array([5, 0, 0, 3, 10])
try:
marks + bonus
except ValueError as error:
print("ValueError:", error)
print("with bonus, as a column:", (marks + bonus[:, np.newaxis]).max(axis=1))
print("np.broadcast_shapes((5, 4), (5, 1)) =", np.broadcast_shapes((5, 4), (5, 1)))Output
top mark per subject: [91 90 93 88] (4,) [[ 85.7 71.1 87.1 79.5] [ 60.4 80. 52.7 100. ] [100. 94.4 82.8 70.5] [ 72.5 64.4 100. 85.2] [ 46.2 100. 73.1 92. ]] (3, 1) + (4,) gives (3, 4) [[ 1 2 3 4] [11 12 13 14] [21 22 23 24]] ValueError: operands could not be broadcast together with shapes (5,4) (5,) with bonus, as a column: [ 86 88 91 96 100] np.broadcast_shapes((5, 4), (5, 1)) = (5, 4)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 broadcasting.py
Step through the same four operations, one pair of sizes at a time:
The drawing shows the whole scene; its 12 steps are listed below.
Every step as text
- From the right. marks has shape (5, 4); top, the best mark of each subject, has shape (4,). NumPy compares shapes from the right, so the 4 subjects of marks meet the 4 values of top: equal sizes fit.
- A missing axis is 1. top has no axis 0. A missing size counts as 1, as if top had the shape (1, 4).
- The 1 stretches. A size of 1 fits any size: it stretches to 5 without copying, so the same four top marks are used for every student.
- The result. Each axis of the result takes the larger size: (5, 4). Every mark is divided by the top mark of its subject, so Chen's 91 in maths becomes 100.0 and Asha's 78 becomes 85.7.
- A column and a row. column has shape (3, 1) and row (4,). From the right, the 1 of column meets the 4 of row, and the 1 stretches to 4.
- Both stretch. On axis 0, row's missing size counts as 1 and stretches to 3. Each array stretches along a different axis.
- A whole table. The result is (3, 4): every value of column added to every value of row, from 1 2 3 4 in the first row to 21 22 23 24 in the last. It does the work of a nested loop.
- One bonus per student. bonus holds five values, one per student, with shape (5,). From the right, its 5 meets the last axis of marks, the 4 subjects.
- Neither is 1. 4 and 5 differ and neither is 1, so NumPy stops with ValueError: operands could not be broadcast together with shapes (5,4) (5,).
- Make it a column. bonus[:, np.newaxis] adds an axis after the first, giving shape (5, 1): one row per student and a single column.
- Now it fits. From the right: the 1 of the column meets the 4 subjects and stretches, so each student's bonus is used for all four marks. Then 5 meets 5, equal.
- The result. The result is (5, 4) again. With each bonus added to all four of the student's marks, the best totals are 86, 88, 91, 96 and 100.
marks.max(axis=0) is a row of four values, the top mark of each subject, so dividing the table by it turns every mark
into a percentage of the best mark in its subject, without a loop. A column of shape (3, 1) and a row of shape (4,)
stretch in both directions into a whole (3, 4) table, which is the vectorised form of a nested loop.
np.broadcast_shapes() gives the result shape of any combination without creating an array; you will write that
function yourself in the practice section.
A value per row lines up with the columns
One bonus per student is five values, but as a (5,) array it meets the last axis, the four subjects, so
marks + bonus fails. Make it a column first: marks + bonus[:, np.newaxis], or bonus.reshape(-1, 1), has shape
(5, 1) and says “one value per row”.
Checkpoint
Check: Broadcasting shapes
A slice is a view; a mask selection is a copy
Slicing a list copies the selected elements into a new list. Slicing an array does not: it returns a A NumPy array that reads the memory of another array instead of holding its own copy, as a slice such as a[2:5] does. Writing through it changes the original., a new array object that reads the same memory. Selecting with a boolean mask or a list of positions is different again, and returns a copy:
"""A slice of an array is a view of the same data; a boolean selection and .copy() are new arrays."""
import numpy as np
readings = np.array([21.5, 22.0, 35.9, 22.4, 21.8, 36.2])
first_three = readings[:3] # a view: no data is copied
first_three[0] = 21.6 # so this changes readings as well
print("after editing the slice: ", readings)
hot = readings[readings > 30] # a copy: a new array holding the selected values
hot[:] = 0.0 # readings does not change
print("after editing the selection:", readings)
backup = readings.copy()
readings[readings > 30] = np.nan # assigning through a mask writes into the original
print("readings:", readings)
print("backup: ", backup)
print("the slice shares memory with readings:", np.shares_memory(first_three, readings))
print("the copy shares memory with readings: ", np.shares_memory(backup, readings))Output
after editing the slice: [21.6 22. 35.9 22.4 21.8 36.2] after editing the selection: [21.6 22. 35.9 22.4 21.8 36.2] readings: [21.6 22. nan 22.4 21.8 nan] backup: [21.6 22. 35.9 22.4 21.8 36.2] the slice shares memory with readings: True the copy shares memory with readings: False
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 views_and_copies.py
Step through the program and follow the arrow from each name to the block of memory it reads:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Six readings. readings is an array of six float64 values, stored side by side in one block of memory.
- A slice is a view. readings[:3] copies nothing. first_three is a new array object that reads the first three values of the same block: a view.
- Writing through the view. first_three[0] = 21.6 writes into block 1, so readings[0] is 21.6 as well. The first line of the output shows it.
- A mask selection copies. readings[readings > 30] picks 35.9 and 36.2 into a new block. hot is a new array that reads only that block.
- Editing the copy. hot[:] = 0.0 zeroes block 2. Block 1 is untouched, so readings still holds 35.9 and 36.2: the second line of the output.
- A copy kept aside. readings.copy() copies all six values into block 3, which no other name reads. That is the raw data, kept before cleaning.
- Assigning through a mask. readings[readings > 30] = np.nan is an assignment, not a selection: it writes NaN into block 1 itself, where the mask is True. Block 3 keeps the raw values.
- Asking NumPy. np.shares_memory() says True for first_three and readings, which share block 1, and False for backup, whose block 3 is its own.
Changing the slice first_three changed readings, because both read block 1, and np.shares_memory() confirms it.
Changing the result of readings[readings > 30] left readings alone, because a mask selection builds a new array.
NumPy’s documentation gives the rule in two parts: basic indexing, such as a slice, always makes a view, and advanced
indexing, with a mask or a list of positions, always makes a copy. Assigning through a mask,
readings[readings > 30] = np.nan, is not a selection at all: the assignment happens in place, in the original. When
you need an independent array, for example to keep the raw data before cleaning it, call copy() first.
A slice taken as a backup
raw = readings[:] would be a copy for a list, but for an array it is a view: cleaning readings afterwards changes
raw as well. Write raw = readings.copy() before you change anything.
Checkpoint
Check: Views and copies
Fixed-size integers wrap around without an error
Python’s int grows as large as a number needs and never overflows. NumPy’s integers are each a
An integer type with a set number of bits, such as NumPy's int32, which holds -2,147,483,648 to 2,147,483,647. A result outside that range wraps around.:
an int32 holds values from −2,147,483,648 to 2,147,483,647, and the program below doubles three daily view counts
stored that way:
"""Fixed-size NumPy integers wrap around without an error; Python ints never overflow."""
import numpy as np
print(np.iinfo(np.int32))
views = np.array([2_000_000_000, 1_500_000_000, 900_000_000], dtype=np.int32)
print("int32 array * 2: ", views * 2) # silently wrong for the first two
print("Python ints * 2: ", [v * 2 for v in views.tolist()])
print("int64 array * 2: ", views.astype(np.int64) * 2)
print("sum of the int32 array:", views.sum(), "as", views.sum().dtype) # sum() widens to int64 here
with np.errstate(over="raise"): # a setting for floating-point errors and integer scalars, not integer arrays
print("int32 array * 2 under errstate(over='raise'):", views * 2)
try:
np.array([3_000_000_000], dtype=np.int32) # a Python int that does not fit is refused
except OverflowError as error:
print("OverflowError:", error)Output
Machine parameters for int32 --------------------------------------------------------------- min = -2147483648 max = 2147483647 --------------------------------------------------------------- int32 array * 2: [ -294967296 -1294967296 1800000000] Python ints * 2: [4000000000, 3000000000, 1800000000] int64 array * 2: [4000000000 3000000000 1800000000] sum of the int32 array: 4400000000 as int64 int32 array * 2 under errstate(over='raise'): [ -294967296 -1294967296 1800000000] OverflowError: Python integer 3000000000 out of bounds for int32
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 int32_overflow.py
Doubling 2,000,000,000 in an int32 array gave −294,967,296: the true answer, 4,000,000,000, minus 2^32. The same
values as Python integers, or as int64, doubled correctly. The documentation of numpy.sum gives the rule: integer
arithmetic is modular, and overflow raises no error. Not even np.errstate(over="raise") caught it: that setting
covers floating-point errors and integer scalars, and under it the array wrapped exactly as before. Step through what
happens to each of the three results:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- The exact result. views * 2 doubles every value of the int32 array. For the first view count, 2,000,000,000, the exact result is 4,000,000,000.
- Too big for 32 bits. An int32 holds at most 2,147,483,647, and 4,000,000,000 is above that. NumPy checks nothing here: there is no error and no warning.
- Wrap around. Integer arithmetic in NumPy is modular. A result above the range loses 2^32: 4,000,000,000 - 4,294,967,296 = -294,967,296.
- Silently wrong. The array holds -294,967,296, the first number the program printed. A count of views has turned negative, and the output says nothing about it.
- The second count. 1,500,000,000 doubles to 3,000,000,000, also above the limit, and wraps to 3,000,000,000 - 4,294,967,296 = -1,294,967,296.
- The third fits. 900,000,000 doubles to 1,800,000,000, which is within the range, so the array keeps it exactly.
- Convert first. After views.astype(np.int64), every value has 64 bits and the range reaches 9,223,372,036,854,775,807, so all three doubles fit: 4,000,000,000, 3,000,000,000 and 1,800,000,000.
Two details soften the trap. sum() of a small integer type adds in the platform’s default integer, here int64,
which is why the total was right, and NumPy 2 refuses to put a Python integer that does not fit into an int32 array
and raises OverflowError. Arithmetic between arrays gets no such check.
The wrapped value can be worked out with Python’s own integers: add or take away 2^32 until the result lies in the
range. This program does exactly that and prints the numbers NumPy printed, and also −1,486,618,624, the value NumPy’s
documentation shows for 100 ** 9 in 32 bits. Press Edit, put a view count of your own into the list, and run it:
"""What an int32 keeps of a result, worked out with Python's own integers: the true value, moved into the range
-2**31 .. 2**31 - 1 by adding or taking away 2**32 as often as needed."""
LOW, HIGH = -(2**31), 2**31 - 1
def as_int32(n):
"""The value a 32-bit signed integer holds after arithmetic that gave n."""
return (n - LOW) % 2**32 + LOW
for views in [2_000_000_000, 1_500_000_000, 900_000_000]:
true = views * 2
kept = as_int32(true)
note = "fits" if LOW <= true <= HIGH else f"{true:,} - {2**32:,}"
print(f"{views:,} * 2 = {true:,} -> int32 holds {kept:,} ({note})")
print("100 ** 9 in 32 bits:", f"{as_int32(100**9):,}")
print("one past the largest int32:", f"{as_int32(HIGH + 1):,}")Output
2,000,000,000 * 2 = 4,000,000,000 -> int32 holds -294,967,296 (4,000,000,000 - 4,294,967,296) 1,500,000,000 * 2 = 3,000,000,000 -> int32 holds -1,294,967,296 (3,000,000,000 - 4,294,967,296) 900,000,000 * 2 = 1,800,000,000 -> int32 holds 1,800,000,000 (fits) 100 ** 9 in 32 bits: -1,486,618,624 one past the largest int32: -2,147,483,648
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 wrap32.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The defence is to choose the dtype on purpose. Keep the default int64 for counts and money in the smallest unit,
check the limits with np.iinfo() when data arrives as int32 or int16 (images and files often do), and convert
before arithmetic that can grow, as astype(np.int64) did above. When totals must be exact beyond 64 bits, Python
integers are the right tool, and for exact decimal amounts the decimal module of the
exact arithmetic lesson is.
Arithmetic on int32 without checking the range
steps * 1_000_000 on an int32 array of step counts is right for small counts and silently wrong, even negative,
for counts above 2,147: the product leaves the range. Convert first, steps.astype(np.int64) * 1_000_000, or keep
such data as int64 from the start.
Checkpoint
Check: Fixed-size integers that wrap around
Labelled, mixed tables belong in pandas
NumPy arrays are the right tool when the data is a grid of numbers of one type: measurements, pixels, matrices, simulations. Real tables are often not like that. They have column names, a mix of text, dates and numbers, missing values, and they arrive as CSV or spreadsheet files. That is the work of pandas, which its documentation describes as data structures for “relational” or “labeled” data, built on top of NumPy. Answer the three questions in the table for data of your own:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Structure | Values | Labels | Arithmetic on every value | Missing values | Reading files |
|---|---|---|---|---|---|
| Python list Holds: any objects, of any types | Good anything, mixed types in any position | Poor positions only | Poor a loop or a comprehension, one value at a time | Mixed whatever you choose, often None | Mixed the csv and json modules, row by row |
| NumPy array Holds: values of one dtype | Poor one dtype for the whole array | Poor positions only | Good vectorised, with broadcasting | Mixed NaN, in floating-point arrays only | Mixed np.loadtxt for plain numeric text |
| pandas DataFrame Holds: named columns of their own dtypes | Good one dtype per column; columns can differ | Good column names and a row index | Good vectorised per column, aligned by labels | Good handled in every column type, not only floats | Good CSV, Excel, JSON, Parquet, SQL databases and more |
How the answers decide
- “Is every value a number of the same type?” No: text, dates and numbers mix: rules out NumPy array. An array has one dtype for every value; text, dates and numbers side by side belong in DataFrame columns, each with its own dtype.
- “Do you look values up by a column name or a row label?” Yes, by names or labels: rules out Python list and NumPy array. Lists and arrays are indexed by position only; a DataFrame has column names and a row index.
- “Will you do arithmetic on many values at once?” Yes, on thousands of values or more: rules out Python list. A list needs a Python loop for every value; arrays and DataFrame columns run the loop in compiled code.
- “Is every value a number of the same type?” Yes, one numeric type and “Do you look values up by a column name or a row label?” No, positions are enough: adds a caveat to pandas DataFrame. A grid of one numeric type, read by position, is what a NumPy array is for; a DataFrame adds labels this data does not need.
- “Will you do arithmetic on many values at once?” No, a handful of values now and then and “Do you look values up by a column name or a row label?” No, positions are enough: adds a caveat to NumPy array and pandas DataFrame. For a handful of values, creating an array costs more than the loop it saves, as the recorded run of small_or_large.py showed.
In short: a list for small or mixed collections, a NumPy array for numeric grids and the maths on them, and a pandas DataFrame for tables of real-world data. If your next step is analysing tables, continue with the Data Analysis with Python track, which uses pandas 3 and also covers polars and DuckDB. Everything in this lesson carries over: a pandas column is backed by an array, and broadcasting and dtypes behave there as they do here.
Checkpoint
Check: NumPy or pandas
Practise: work out a broadcast shape
The exercise asks for broadcast_shape(*shapes), the rule of this lesson written without NumPy: compare the shapes
from the right, let sizes of 1 stretch, and raise ValueError when two sizes differ and neither is 1. Its sample tests
run in your browser:
Exercise · Medium · Python
Work out the shape that broadcasting gives
Write broadcast_shape(*shapes) in shapes.py without importing NumPy. It takes any number of shapes, each a tuple of non-negative integers such as (5, 4), (4,) or (), and returns the shape of the result that NumPy's broadcasting rule gives them, as a tuple.
- Compare the shapes from the right, one axis at a time. A shape with fewer dimensions counts as if it had sizes of 1 on the left.
- On each axis the sizes must be equal or 1. The result takes the size that is not 1 (or 1, when every size is 1). A size of 0 follows the same rule: it fits with 0 and with 1, and the result is 0.
- Raise
ValueErrorwhen any axis has two different sizes that are both not 1. - With no shapes at all, return
().
For example, broadcast_shape((5, 4), (4,)) is (5, 4), broadcast_shape((3, 1), (4,)) is (3, 4), and broadcast_shape((5, 4), (5,)) raises ValueError.
Starter code · shapes.py
def broadcast_shape(*shapes):
"""Return the shape that NumPy's broadcasting rule gives these shapes, or raise ValueError."""
# Replace these lines with your code.
return shapes[0] if shapes else ()The sample tests · test_shapes.py
from shapes import broadcast_shape
def refused(*shapes):
try:
broadcast_shape(*shapes)
except ValueError:
return True
return False
def test_equal_shapes():
"""two equal shapes give the same shape"""
assert broadcast_shape((5, 4), (5, 4)) == (5, 4)
assert broadcast_shape((7,), (7,)) == (7,)
def test_row_against_table():
"""a shorter shape is padded with 1s on the left, in either order"""
assert broadcast_shape((5, 4), (4,)) == (5, 4)
assert broadcast_shape((4,), (5, 4)) == (5, 4)
assert broadcast_shape((6, 5, 4), (5, 1)) == (6, 5, 4)
def test_column_against_row():
"""sizes of 1 stretch in both arrays"""
assert broadcast_shape((3, 1), (4,)) == (3, 4)
assert broadcast_shape((2, 1, 4, 1), (3, 1, 5)) == (2, 3, 4, 5)
def test_incompatible_shapes():
"""two different sizes that are not 1 raise ValueError"""
assert refused((5, 4), (5,)), "(5, 4) and (5,) do not broadcast"
assert refused((2, 3), (3, 2)), "(2, 3) and (3, 2) do not broadcast"
assert refused((1, 2), (3, 1), (4, 1)), "3 and 4 meet on the first axis"
def test_scalars_and_many_shapes():
"""() is a scalar, no shapes give (), and any number of shapes combine"""
assert broadcast_shape((), (3,)) == (3,)
assert broadcast_shape() == ()
assert broadcast_shape((1, 3), (2, 1), (1,)) == (2, 3)
def test_size_zero():
"""0 fits with 0 and with 1, never with another size"""
assert broadcast_shape((0,), (1,)) == (0,)
assert broadcast_shape((2, 0), (1, 0)) == (2, 0)
assert refused((0,), (3,)), "0 and 3 do not broadcast"A hint
Work out how many dimensions the result has first: the length of the longest shape. Then loop over the axes from the right with for i in range(1, ndim + 1) and look at shape[-i] in every shape that is long enough. Start each axis at size 1; a size of 1 changes nothing, a first other size becomes the axis's size, and any later size that differs from it is an error. Collect the sizes and reverse them at the end.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
After the exercise, run through the seven ideas once more, from indexing to pandas. The questions begin with the idea your checkpoints found shakiest, and a wrong pick is answered with the slip behind it:
Practice that adapts to you
Practise this lesson
20 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
What is the difference between a NumPy array and a Python list?
A list is a sequence of references to Python objects of any type, and it can grow and shrink. A NumPy array is a block
of memory holding values of one dtype, such as int64 or float64, side by side, and its size is fixed when it is
created. That layout lets NumPy apply an operation to every element in compiled code, so a * 2 doubles a million
numbers without a Python loop and is usually tens of times faster than a comprehension, and it lets arrays of
different shapes combine through broadcasting. Arrays also behave differently: arithmetic on them is element-wise
([1, 2] * 2 repeats a list, while an array doubles its values), slicing returns a view instead of a copy, and
fixed-size integer types can overflow. Use lists for small or mixed collections, arrays for large numeric data.
Key takeaways
- An
ndarrayholds values of one dtype in one block of memory, read with one position or slice per axis and selected with boolean masks. - A reduction removes the axis you pass:
axis=0gives one result per column,axis=1one per row, and no axis one value for the whole array. - Whole-array operations run their loop in compiled code, more than ten times faster than a Python loop in the recorded runs, but not for a handful of values or when every call converts a list to an array and back.
- Broadcasting compares shapes from the right, so sizes must be equal or 1, a missing size counts as 1, and a value
per row needs the shape
(n, 1). - Slices are views of the same memory, while mask selections and
copy()make new arrays, and assigning through a mask writes into the original. int32and smaller types wrap around silently, so totals belong inint64or Python’sint, and labelled, mixed tables belong in pandas.
References
- NumPy: the absolute basics for beginners (NumPy Developers)
- What is NumPy? (NumPy Developers)
- Universal functions (ufunc) basics (NumPy Developers)
- Broadcasting (NumPy Developers)
- Data types: overflow errors (NumPy Developers)
- numpy.sum (NumPy Developers)
- numpy.seterr (NumPy Developers)
- Copies and views (NumPy Developers)
- numpy.shares_memory (NumPy Developers)
- numpy.transpose (NumPy Developers)
- numpy.vectorize (NumPy Developers)
- pandas: package overview (pandas development team)
- pandas: IO tools (text, CSV, HDF5 and more) (pandas development team)
- Integers: int, and the unlimited precision of Python integers (Python Software Foundation)
- Design and History FAQ: how are lists implemented in CPython? (Python Software Foundation)
- The Python Language Reference: data model (Python Software Foundation)
- timeit: measure execution time of small code snippets (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress