NumPy essentials for analysts

NumPy is the Python library for arrays: blocks of values of one type, stored side by side in memory, with operations that run over the whole block at once in compiled code. Analysts use it constantly, often without typing import numpy, because for the most part pandas keeps each numeric column of a DataFrame as a NumPy array (pandas user guide).

Four things about arrays decide whether column calculations are fast and right: why whole-array operations are fast, how an array chooses its type, where integers overflow without a word, and how to write an if-else rule for a whole column at once. This lesson takes each in turn on NumPy 2.5.3 and ends with random data that you, and anyone else, can reproduce.

One type, side by side in memory

A Python list holds references, and each reference points to a separate object that carries its own type and size. A NumPy array holds plain values of a single type, its dtype, packed next to each other:

Because every value has the same type and size, NumPy can run one loop, written in C, over the whole block, without asking each value what it is. That is what vectorised means: you write one operation on whole arrays, such as quantity * price, and NumPy applies it to every element. An array also knows its shape (rows and columns), its dtype and its size in bytes, nbytes, which is simply the number of values times the size of one.

How much faster, measured

This program computes 100,000 order totals, quantity times price, in three ways. It first checks that all three give exactly the same values, then times each one with timeit and keeps its best round:

# Order totals (quantity × price) for 100,000 orders, three ways, timed with timeit.
import platform
import timeit

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
quantity = rng.integers(1, 6, size=100_000)  # 1 to 5 items
price = rng.integers(20, 500, size=100_000) * 1.0  # 20 to 499 a piece
q_list, p_list = quantity.tolist(), price.tolist()  # the same numbers as Python lists
orders = pd.DataFrame({"quantity": quantity, "price": price})


def pandas_apply():
    return orders.apply(lambda row: row["quantity"] * row["price"], axis=1)


def python_loop():
    totals = []
    for i in range(len(q_list)):
        totals.append(q_list[i] * p_list[i])
    return totals


def vectorised():
    return quantity * price


assert pandas_apply().tolist() == python_loop() == vectorised().tolist()  # all three give the same values
ways = [("pandas apply, axis=1", pandas_apply, 1, 3), ("Python loop", python_loop, 20, 7), ("NumPy vectorised", vectorised, 20, 7)]
best = {}
for name, fn, number, rounds in ways:
    best[name] = min(timeit.repeat(fn, number=number, repeat=rounds)) / number  # seconds per call, best round
    print(f"{name:<21} {best[name] * 1000:9.3f} ms")
loop, apply = best["Python loop"], best["pandas apply, axis=1"]
print(f"vectorised: {loop / best['NumPy vectorised']:.0f} times faster than the loop, {apply / best['NumPy vectorised']:.0f} times faster than apply")
print(f"CPython {platform.python_version()}, NumPy {np.__version__}, pandas {pd.__version__}, {platform.machine()}")

pandas apply, axis=1   1078.492 ms
Python loop               6.400 ms
NumPy vectorised          0.083 ms
vectorised: 77 times faster than the loop, 12953 times faster than apply
CPython 3.14.8, NumPy 2.5.3, pandas 3.0.6, arm64

In the run recorded here, on a Mac with an Apple processor, the vectorised version took 0.083 ms, the Python loop 6.4 ms and pandas apply with axis=1 just over a second: 77 and about 13,000 times slower. Your times will differ, and so will the next run’s, because they depend on the computer and on what else it is doing, but the gaps stay enormous. apply with axis=1 is the slowest because it builds a small Series for every row and calls your Python function on it, 100,000 times. A list comprehension writes the loop more compactly, yet it still handles one Python object at a time, so it lands near the loop, nowhere near NumPy.

What type will the array have?

NumPy chooses one dtype that can hold every value you give it. Predict each line before you read the output:

# What dtype does NumPy pick? Six small arrays, then the same missing value in pandas.
import numpy as np
import pandas as pd

examples = {
    "[3, 5, 8]": np.array([3, 5, 8]),
    "[3, 5, 8.5]": np.array([3, 5, 8.5]),
    "[3, 5, np.nan]": np.array([3, 5, np.nan]),
    "[3, 5, None]": np.array([3, 5, None]),
    "[True, False, 1]": np.array([True, False, 1]),
    '["3", 5, 8]': np.array(["3", 5, 8]),
}
for text, arr in examples.items():
    print(f"np.array({text:<16}) -> {str(arr.dtype):<8} {arr}")

print()
print("int64 uses", np.array([3, 5, 8]).itemsize, "bytes per value; a million values take",
      np.zeros(1_000_000, dtype=np.int64).nbytes // 1_000_000, "MB")
print(pd.Series([3, 5, None]).dtype, pd.Series([3, 5, None]).tolist())
print(pd.Series([3, 5, None], dtype="Int64").dtype, pd.Series([3, 5, None], dtype="Int64").tolist())

np.array([3, 5, 8]       ) -> int64    [3 5 8]
np.array([3, 5, 8.5]     ) -> float64  [3.  5.  8.5]
np.array([3, 5, np.nan]  ) -> float64  [ 3.  5. nan]
np.array([3, 5, None]    ) -> object   [3 5 None]
np.array([True, False, 1]) -> int64    [1 0 1]
np.array(["3", 5, 8]     ) -> <U21     ['3' '5' '8']

int64 uses 8 bytes per value; a million values take 8 MB
float64 [3.0, 5.0, nan]
Int64 [3, 5, <NA>]

• Whole numbers become int64, 8 bytes each, so a million of them take 8 MB.
• One decimal turns the whole array into float64.
• np.nan, the usual marker for a missing number, is itself a float, so a missing value turns a column of whole numbers into float64: 3 and 5 become 3.0 and 5.0.
• None has no number type at all, so the array falls back to object: an array of references to Python objects, which is as slow as a list.
• True and False become 1 and 0 next to an integer.
• One string turns every value into a string (<U21 means Unicode text of up to 21 characters), so '3', '5' and '8' are no longer numbers and arithmetic on them fails.

pandas inherits the missing-value rule: the same three values give a float64 column. Its nullable Int64 dtype (with a capital I) keeps the integers and marks the gap as <NA> instead (nullable integers).

Integers that wrap around

A Python int grows as large as it needs to and never overflows. NumPy’s integers have a fixed width, and when a result does not fit, it wraps around (data types):

# Fixed-width integers wrap around when a result does not fit.
import warnings

import numpy as np

small = np.array([120, 127], dtype=np.int8)  # int8 holds -128 to 127
print("int8 array + 1:", small + 1)  # wraps around, no warning

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    one = np.int8(127) + np.int8(1)  # a single value does warn
print("int8 scalar + 1:", one, "| warning:", caught[0].message)

quantity = np.array([3, 40_000], dtype=np.int32)  # items on each order line
price = np.array([19_900, 75_000], dtype=np.int32)  # price of one item, in paise
print("int32 line totals:", quantity * price)  # 40,000 x 75,000 = 3,000,000,000
print("int64 line totals:", quantity.astype(np.int64) * price)
print("largest int32:", np.iinfo(np.int32).max)
print("summing int32 values accumulates in", np.array([1, 2], dtype=np.int32).sum().dtype)

int8 array + 1: [ 121 -128]
int8 scalar + 1: -128 | warning: overflow encountered in scalar add
int32 line totals: [      59700 -1294967296]
int64 line totals: [     59700 3000000000]
largest int32: 2147483647
summing int32 values accumulates in int64

• An int8 holds −128 to 127, so 127 + 1 in an array becomes −128, with no warning. The same sum on a single NumPy value does warn: overflow encountered in scalar add.
• The int32 line totals are the dangerous case. 40,000 items at 75,000 paise (or cents) each make 3,000,000,000, more than the largest int32 (2,147,483,647), so the line total comes out as −1,294,967,296. Converting one column to int64 before multiplying gives the right total.
• sum() is safer: for integer types smaller than the platform’s default integer, NumPy adds them up in that default integer, which is 64-bit on 64-bit computers (numpy.sum). Products and differences of two int32 columns get no such help.

The default integer has not always been 64-bit everywhere: before NumPy 2.0 it was 32-bit on Windows (NumPy 2.0.0 release notes), so the same code could overflow on a colleague’s Windows computer and not on yours. Keep counts and money in int64 (amounts as whole paise or cents) or in float64, check the limits with np.iinfo, and convert before you multiply.

If-else for a whole column

A delivery fee rule makes a good test: no fee for orders from 500 up, 25 for orders from 200 to below 500, and 40 for anything smaller. A loop would test each order in turn. NumPy does it with conditions on whole arrays:

# Delivery fees by order value, without a loop: no fee from 500, 25 from 200, 40 below that.
import numpy as np

amount = np.array([120.0, 199.0, 199.5, 200.0, 350.0, 499.0, 500.0, np.nan])  # the last value is missing

fee = np.select([amount >= 500, amount >= 200], [0, 25], default=40)
print("amount:     ", amount)
print("fee:        ", fee)

wrong = np.select([amount >= 200, amount >= 500], [25, 0], default=40)  # the first true condition wins
print("wrong order:", wrong)

safe = np.select([np.isnan(amount), amount >= 500, amount >= 200], [np.nan, 0, 25], default=40)
print("NaN handled:", safe)

big = amount >= 500  # a boolean mask: one True or False per order
print("orders of 500 or more:", big.sum(), "of", amount.size, f"({big.mean():.1%})")
print("their values:", amount[big])
print("np.where, two bands:", np.where(amount >= 500, 0, 40))

amount:      [120.  199.  199.5 200.  350.  499.  500.    nan]
fee:         [40 40 40 25 25 25  0 40]
wrong order: [40 40 40 25 25 25 25 40]
NaN handled: [40. 40. 40. 25. 25. 25.  0. nan]
orders of 500 or more: 1 of 8 (12.5%)
their values: [500.]
np.where, two bands: [40 40 40 40 40 40  0 40]

The pieces, in the order the program uses them:

• np.select(conditions, choices, default) handles a rule with several bands. For each element it takes the choice of the first condition that is true, and the default when none is (numpy.select). Test the exact boundaries: 199.5 pays 40 because the rule says “from 200”, while 200 pays 25 and 500 pays nothing.
• The order of the conditions is part of the rule. In the “wrong order” line, amount >= 200 comes first and is also true for 500, so that order pays 25 instead of nothing. Put the narrowest band first.
• A missing amount falls through to the default. Every comparison with nan is false, so the order without an amount was charged the highest fee, 40, without any error. Putting np.isnan(amount) first, with np.nan as its choice, keeps a missing value missing.
• A boolean mask such as amount >= 500 is an array of True and False, one per order. Its sum() counts the orders that match, its mean() gives their share, and amount[mask] selects them.
• np.where(condition, if_true, if_false) is the two-band version: one condition, two outcomes (numpy.where).

Random data you can repeat

Simulations, samples and practice tables need random numbers that come out the same every time the analysis runs. Make them from a Generator created with a seed:

# Random numbers you can repeat: a Generator made from a seed.
import numpy as np

rng = np.random.default_rng(42)
print("five dice, seed 42:   ", rng.integers(1, 7, size=5))
print("then two uniforms:    ", rng.random(2).round(4))

again = np.random.default_rng(42)  # a new Generator with the same seed starts the same stream
print("same seed, same dice: ", again.integers(1, 7, size=5))
print("seed 43:              ", np.random.default_rng(43).integers(1, 7, size=5))

baskets = np.random.default_rng(2026).lognormal(mean=5.5, sigma=0.6, size=1_000).round(2)
print("1,000 practice baskets: min", baskets.min(), "| median", np.median(baskets), "| max", baskets.max())

five dice, seed 42:    [1 5 4 3 3]
then two uniforms:     [0.6974 0.0942]
same seed, same dice:  [1 5 4 3 3]
seed 43:               [4 4 3 1 4]
1,000 practice baskets: min 25.83 | median 250.075 | max 1705.71

np.random.default_rng(42) returns a Generator whose numbers depend only on the seed, so a second Generator made with 42 repeats the first one’s dice exactly, and seed 43 gives different ones. Each Generator is an object that belongs to your code. The older style, np.random.seed(42) followed by functions such as np.random.randint, sets one hidden state that every part of a program shares, so any library that draws a random number shifts your sequence; NumPy keeps it only for backward compatibility (Random Generator).

A seed is not the whole story. NumPy’s guarantee of a repeatable stream holds only while nothing else changes: the NumPy build, the rest of the software around it and the hardware all have to match, and a new feature release is allowed to change what a Generator method returns (compatibility policy). That is one more reason to pin NumPy in the lock file, as the previous lesson did. The last line of the example draws 1,000 skewed values with a log-normal distribution, a shape close to real order values: most are small, a few are large, and the median, 250.075, sits far below the largest, 1,705.71.

Check yourself

Interview questions

Fresher · Why does a missing value turn a NumPy array of integers into float64?

A NumPy array has one dtype for all its elements, and the integer dtypes have no value left over that could mean “missing”. The usual marker for a missing number, np.nan, is a floating-point value, so an array that must hold 3, 5 and nan needs a type that can store all three, and NumPy picks float64. The whole numbers become 3.0 and 5.0, which looks harmless but changes how they print, compare and join, and whole numbers above 2⁵³ can no longer all be stored exactly. pandas behaves the same way for its default int64 columns, while its nullable Int64 dtype keeps the integers and marks the gap separately, as <NA>.

Key takeaways

• An array holds one dtype in one block of memory, so a vectorised operation runs as one compiled loop: in the run above it beat a Python loop and pandas apply by orders of magnitude.
• NumPy picks a dtype that fits every value: a missing value makes integers float64, None makes object, and one string makes everything text.
• Fixed-width integers wrap around silently in arrays; multiply in int64 and check limits with np.iinfo.
• np.where handles two bands and np.select many; the first true condition wins, and nan falls through to the default unless you test for it first.
• Make random data with np.random.default_rng(seed) and pin the NumPy version to repeat it exactly.