Reshaping, stacking and splitting arrays

Three years of monthly electricity use arrive as one row of 36 numbers, and the quarterly totals are a single line away: kwh.reshape(3, 4, 3).sum(axis=2). Reshaping reads the values in one order and lays them into a new shape without moving them, which is why it is instant, and also why reading data in the wrong order raises no error.

You will reshape with -1 in C or F order, move, add and remove axes, join arrays with concatenate, stack and their shortcuts, split them back, repeat values with or without copying, and replace the calls that NumPy 2.5 deprecates or removes.

reshape keeps the order of the values and works out one length

Here are the 36 readings, January to December of three years, read as years and then as quarters:

"""Three years of monthly electricity use: one row of 36 numbers read as years, quarters and months."""
import numpy as np

kwh = np.array([
    310, 280, 260, 230, 250, 340, 380, 370, 300, 260, 270, 320,  # year 1, January to December
    330, 290, 270, 240, 260, 350, 400, 390, 310, 270, 280, 340,  # year 2
    320, 300, 280, 250, 270, 360, 410, 400, 320, 280, 290, 350,  # year 3
], dtype=np.int64)
years = kwh.reshape(3, 12)  # the same 36 values, read as 3 rows of 12
print("years:", years.shape, "| a view of kwh:", np.shares_memory(years, kwh))
quarters = years.reshape(3, 4, 3)  # each year as 4 quarters of 3 months
print("quarterly totals:\n", quarters.sum(axis=2))
print("yearly totals:", years.sum(axis=1))
print("reshape(-1, 12):", kwh.reshape(-1, 12).shape, "| reshape(6, -1):", kwh.reshape(6, -1).shape)
for shape in [(5, -1), (-1, -1)]:
    try:
        kwh.reshape(shape)
    except ValueError as error:
        print(f"reshape{shape}: ValueError: {error}")
six = np.arange(6, dtype=np.int64)
print("order C:", six.reshape(2, 3).tolist(), "| order F:", six.reshape(2, 3, order="F").tolist())
print("F order read back in F order:", six.reshape(2, 3, order="F").reshape(-1, order="F").tolist())

years: (3, 12) | a view of kwh: True
quarterly totals:
 [[ 850  820 1050  850]
 [ 890  850 1100  890]
 [ 900  880 1130  920]]
yearly totals: [3570 3730 3830]
reshape(-1, 12): (3, 12) | reshape(6, -1): (6, 6)
reshape(5, -1): ValueError: cannot reshape array of size 36 into shape (5,newaxis)
reshape(-1, -1): ValueError: can only specify one unknown dimension
order C: [[0, 1, 2], [3, 4, 5]] | order F: [[0, 2, 4], [1, 3, 5]]
F order read back in F order: [0, 1, 2, 3, 4, 5]

kwh.reshape(3, 12) lays the values into 3 rows of 12, one year a row, and it is a view: no value moved, as np.shares_memory shows. years.reshape(3, 4, 3) reads each year as 4 quarters of 3 months, so summing the last axis gives the quarterly totals, 850, 820, 1,050 and 850 kWh in year 1. One length may be -1, which reshape works out from the size: (-1, 12) became (3, 12) and (6, -1) became (6, 6). A size that does not divide, as in (5, -1), and two unknown lengths both raise a ValueError.

The order is the one thing reshape must be told. In the default C orderC order: Row-major layout: the last index changes fastest in memory, so the values of one row sit side by side. NumPy makes new arrays in C order unless asked otherwise. More in “The ndarray: shape, dtype, strides and memory” the last index changes fastest; with order="F", the F orderF order: Column-major (Fortran) layout: the first index changes fastest in memory, so the values of one column sit side by side. A transpose of a C-order array is in F order. More in “The ndarray: shape, dtype, strides and memory” of Fortran and MATLAB, the first index does. Step through six values in each order:

np.arange(6).reshape(2, 3, order="F") gave [[0, 2, 4], [1, 3, 5]], and reading it back in F order gave 0 to 5 again. NumPy’s reference adds that C and F here “only refer to the order of indexing”, not to how the array happens to sit in memory.

Move, add and remove axes without copying

Files and most image code store a picture as (height, width, channels), while many deep-learning libraries expect (channels, height, width). Moving an axis is a view, and only the stridesstrides: The number of bytes to move in memory to reach the next element along each axis. A (3, 4) int64 array in C order has strides (32, 8). More in “The ndarray: shape, dtype, strides and memory” change. Here is a tiny RGB image, 4 pixels high and 6 wide:

"""A tiny RGB image moved to channels-first, and axes of length 1 added and removed."""
import numpy as np

img = np.arange(4 * 6 * 3, dtype=np.uint8).reshape(4, 6, 3)  # height 4, width 6, 3 colour channels
chw = np.moveaxis(img, -1, 0)  # channels first, the layout many deep-learning libraries expect
print("img", img.shape, "strides", img.strides, "| moveaxis(img, -1, 0)", chw.shape, "strides", chw.strides)
print("moveaxis gives a view:", np.shares_memory(img, chw))
print("np.transpose(img, (2, 0, 1)):", np.transpose(img, (2, 0, 1)).shape, "| np.swapaxes(img, 0, 2):", np.swapaxes(img, 0, 2).shape)
red = img[:, :, 0]
print("red:", red.shape, "| np.expand_dims(red, -1):", np.expand_dims(red, -1).shape)
batch = np.expand_dims(chw, 0)  # a batch that holds one image
print("batch:", batch.shape, "| np.squeeze(batch):", np.squeeze(batch).shape, "| np.squeeze(batch, axis=0):", np.squeeze(batch, axis=0).shape)
try:
    np.squeeze(batch, axis=1)
except ValueError as error:
    print("np.squeeze(batch, axis=1): ValueError:", error)

img (4, 6, 3) strides (18, 3, 1) | moveaxis(img, -1, 0) (3, 4, 6) strides (1, 18, 3)
moveaxis gives a view: True
np.transpose(img, (2, 0, 1)): (3, 4, 6) | np.swapaxes(img, 0, 2): (3, 6, 4)
red: (4, 6) | np.expand_dims(red, -1): (4, 6, 1)
batch: (1, 3, 4, 6) | np.squeeze(batch): (3, 4, 6) | np.squeeze(batch, axis=0): (3, 4, 6)
np.squeeze(batch, axis=1): ValueError: cannot select an axis to squeeze out which has size not equal to one

np.moveaxis(img, -1, 0) moved the channel axis to the front, shape (3, 4, 6), with strides (1, 18, 3): the same bytes read in a new order. NumPy’s reference says of it that the “other axes remain in their original order”. np.transpose(img, (2, 0, 1)) takes the full new order of axes and gives the same result, while np.swapaxes(img, 0, 2) exchanges two axes and gives (3, 6, 4), with height and width swapped as well.

np.expand_dims adds an axis of length 1, as np.newaxis does: the red channel (4, 6) became (4, 6, 1), and one image became a batch of one, (1, 3, 4, 6). np.squeeze removes axes of length 1, and when you name the axis it refuses one that is not of length 1, as the ValueError for axis 1 showed.

concatenate joins along an axis, stack adds one

Joining makes a new array and copies the inputs into it. Two functions do the work and the others are shortcuts: np.concatenate makes an axis the inputs already have longer, and np.stack joins them, in the words of its reference, “along a new axis”. The run joins two vectors of three values in every way:

"""Joining arrays: concatenate along an axis that exists, stack along a new one, and the shortcuts."""
import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
for label, joined in [
    ("np.concatenate([a, b])", np.concatenate([a, b])),
    ("np.stack([a, b])", np.stack([a, b])),
    ("np.stack([a, b], axis=1)", np.stack([a, b], axis=1)),
    ("np.vstack([a, b])", np.vstack([a, b])),
    ("np.hstack([a, b])", np.hstack([a, b])),
    ("np.column_stack([a, b])", np.column_stack([a, b])),
]:
    print(f"{label:<25} {str(joined.shape):<7} {joined.tolist()}")
m = np.ones((2, 3), dtype=np.int64)
print("2-D, axis=0:", np.concatenate([m, 2 * m]).shape, "| axis=1:", np.concatenate([m, 2 * m], axis=1).shape)
try:
    np.concatenate([a, m])
except ValueError as error:
    print("np.concatenate([a, m]): ValueError:", error)
block = np.block([[m, np.zeros((2, 1), dtype=np.int64)], [np.zeros((1, 3), dtype=np.int64), np.ones((1, 1), dtype=np.int64)]])
print("np.block:", block.shape)
print(block)
print("a join copies its parts:", not np.shares_memory(np.concatenate([a, b]), a))

np.concatenate([a, b])    (6,)    [1, 2, 3, 4, 5, 6]
np.stack([a, b])          (2, 3)  [[1, 2, 3], [4, 5, 6]]
np.stack([a, b], axis=1)  (3, 2)  [[1, 4], [2, 5], [3, 6]]
np.vstack([a, b])         (2, 3)  [[1, 2, 3], [4, 5, 6]]
np.hstack([a, b])         (6,)    [1, 2, 3, 4, 5, 6]
np.column_stack([a, b])   (3, 2)  [[1, 4], [2, 5], [3, 6]]
2-D, axis=0: (4, 3) | axis=1: (2, 6)
np.concatenate([a, m]): ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 1 has 2 dimension(s)
np.block: (3, 4)
[[1 1 1 0]
 [1 1 1 0]
 [0 0 0 1]]
a join copies its parts: True

concatenate gave one vector of 6 values. stack gave (2, 3), one row for each input, or (3, 2) with axis=1. vstack turns 1-D inputs into rows before joining them along the first axis, column_stack turns them into columns, and hstack joins 1-D inputs end to end, like concatenate. concatenate needs inputs with the same number of axes, so a (3,) vector and a (2, 3) table failed. np.block builds an array from nested lists of blocks, joining each inner list side by side and then the rows of blocks, here a (3, 4) array from four blocks. None of the results shares memory with its parts.

Three questions about your arrays decide which function to use. Answer them in the table:

In short: concatenate to make an axis longer, stack to add one, vstack or column_stack to turn vectors into rows or columns, and np.block for a grid of blocks.

Splitting cuts an array back into views

Splitting undoes a join, and it copies nothing: each part is a slice of the original. The run cuts the image into its colour channels, cuts a vector at chosen positions, and cuts one into unequal parts:

"""Splitting undoes a join: equal parts with split, unequal ones with array_split, or at chosen positions."""
import numpy as np

img = np.arange(4 * 6 * 3, dtype=np.uint8).reshape(4, 6, 3)
r, g, b = np.split(img, 3, axis=2)  # three (4, 6, 1) pieces, one per colour
print("np.split(img, 3, axis=2):", r.shape, g.shape, b.shape, "| views of img:", np.shares_memory(r, img))
print("concatenated back, equal to img:", np.array_equal(np.concatenate([r, g, b], axis=2), img))
x = np.arange(10)
print("np.split(x, [3, 7]):", [part.tolist() for part in np.split(x, [3, 7])])
print("np.array_split(np.arange(7), 3):", [part.tolist() for part in np.array_split(np.arange(7), 3)])
try:
    np.split(np.arange(7), 3)
except ValueError as error:
    print("np.split(np.arange(7), 3): ValueError:", error)

np.split(img, 3, axis=2): (4, 6, 1) (4, 6, 1) (4, 6, 1) | views of img: True
concatenated back, equal to img: True
np.split(x, [3, 7]): [[0, 1, 2], [3, 4, 5, 6], [7, 8, 9]]
np.array_split(np.arange(7), 3): [[0, 1, 2], [3, 4], [5, 6]]
np.split(np.arange(7), 3): ValueError: array split does not result in an equal division

np.split(img, 3, axis=2) gave three (4, 6, 1) pieces, one per colour, each a viewview: An array that reads another array's memory instead of holding its own copy, such as a slice or a transpose. A change made through a view changes the original. More in “The ndarray: shape, dtype, strides and memory” of img, and concatenating them back gave an array equal to img. A list of positions says where to cut: [3, 7] gives x[:3], x[3:7] and x[7:]. A number of sections asks for equal parts, and 7 values do not make 3 of them, hence the ValueError. np.array_split accepts any count: NumPy’s reference says it makes n % sections parts of size n // sections + 1 and the rest of size n // sections, so 7 values became parts of 3, 2 and 2. np.vsplit, np.hsplit and np.dsplit are split along axes 0, 1 and 2, except that hsplit cuts a 1-D array along its only axis.

Repeating values: tile, repeat or a read-only view

Three functions repeat values, and they differ in what they repeat and in the memory they take:

"""tile repeats the whole array, repeat repeats each element, and broadcast_to repeats without copying."""
import numpy as np

t = np.array([1, 2, 3], dtype=np.int64)
print("np.tile(t, 2):", np.tile(t, 2).tolist(), "| np.repeat(t, 2):", np.repeat(t, 2).tolist())
print("np.tile(t, (2, 1)):", np.tile(t, (2, 1)).tolist())
print("np.repeat(t, [3, 0, 1]):", np.repeat(t, [3, 0, 1]).tolist())
tiled = np.tile(t, (1000, 1))
view = np.broadcast_to(t, (1000, 3))
print("np.tile(t, (1000, 1)):", tiled.shape, "with", tiled.nbytes, "bytes of its own")
print("np.broadcast_to(t, (1000, 3)):", view.shape, "strides", view.strides, "| reads t's 24 bytes:", np.shares_memory(view, t))
print("its nbytes still says", view.nbytes, "| writeable:", view.flags.writeable)
try:
    view[0, 0] = 9
except ValueError as error:
    print("view[0, 0] = 9: ValueError:", error)

np.tile(t, 2): [1, 2, 3, 1, 2, 3] | np.repeat(t, 2): [1, 1, 2, 2, 3, 3]
np.tile(t, (2, 1)): [[1, 2, 3], [1, 2, 3]]
np.repeat(t, [3, 0, 1]): [1, 1, 1, 3]
np.tile(t, (1000, 1)): (1000, 3) with 24000 bytes of its own
np.broadcast_to(t, (1000, 3)): (1000, 3) strides (0, 8) | reads t's 24 bytes: True
its nbytes still says 24000 | writeable: False
view[0, 0] = 9: ValueError: assignment destination is read-only

np.tile repeats the whole array and np.repeat each element, with one count per element if you give a list, and both write every repeat into new memory: np.tile(t, (1000, 1)) owns 24,000 bytes. np.broadcast_to(t, (1000, 3)) has the same shape but a stride of 0 along the new axis, so all 1,000 rows read the same 24 bytes of t. NumPy’s reference calls it “a readonly view on the original array”, a read-only viewread-only view: A view that NumPy refuses to write through, such as the result of np.broadcast_to, where many positions read the same memory. Its flags.writeable is False, and a write raises ValueError., and writing to it raised ValueError: assignment destination is read-only. Its nbytes still reports 24,000, the positions it covers rather than memory it owns.

What NumPy 2.5 deprecates, and what to write instead

Four releases changed the calls of this lesson, each first with a deprecationdeprecation: A notice that a feature will be removed in a later release. NumPy warns with a DeprecationWarning while the feature still works, then removes it some releases later, as it did with np.row_stack., a warning that the old form will go, and later with its removal. Walk through them in order:

The run tries each old form on the pinned NumPy 2.5.3 and shows what replaces it:

"""What NumPy 2.5 deprecates or removes around reshaping and stacking, and what replaces each."""
import warnings

import numpy as np

print("NumPy", np.__version__)


def warnings_of(change):
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        change()
    return [f"{w.category.__name__}: {str(w.message).splitlines()[0]}" for w in caught]


x = np.arange(6, dtype=np.int64)
print("x.shape = (2, 3):", warnings_of(lambda: setattr(x, "shape", (2, 3))))
print("  instead np.reshape(x, (2, 3)):", np.reshape(np.arange(6), (2, 3)).shape)
y = np.arange(6, dtype=np.int64)
print("y.dtype = np.uint64:", warnings_of(lambda: setattr(y, "dtype", np.uint64)))
print("  instead y.view(np.uint64):", np.arange(6, dtype=np.int64).view(np.uint64).dtype)
z = np.arange(6, dtype=np.int64)
print("z.resize((2, 3)) in place:", warnings_of(lambda: z.resize((2, 3))))
print("  instead np.resize(np.arange(4), (2, 3)):", np.resize(np.arange(4), (2, 3)).tolist())
try:
    np.row_stack
except AttributeError as error:
    print("np.row_stack:", error, "| instead np.vstack")
try:
    np.reshape(np.arange(6), newshape=(2, 3))
except TypeError as error:
    print("np.reshape(..., newshape=...):", error, "| instead shape=")

NumPy 2.5.3
x.shape = (2, 3): ['DeprecationWarning: Setting the shape on a NumPy array has been deprecated in NumPy 2.5.']
  instead np.reshape(x, (2, 3)): (2, 3)
y.dtype = np.uint64: ['DeprecationWarning: Setting the dtype on a NumPy array has been deprecated in NumPy 2.5.']
  instead y.view(np.uint64): uint64
z.resize((2, 3)) in place: []
  instead np.resize(np.arange(4), (2, 3)): [[0, 1, 2], [3, 0, 1]]
np.row_stack: module 'numpy' has no attribute 'row_stack' | instead np.vstack
np.reshape(..., newshape=...): reshape() got an unexpected keyword argument 'newshape' | instead shape=

Assigning to x.shape and to y.dtype both warned with a DeprecationWarning. The release notes give the reason, that changing an array in place is unsafe when it is shared, and the replacements: np.reshape, with copy=False when no copy is allowed, and arr.view(dtype). Both make a new header and leave every other reference to the array as it was. z.resize((2, 3)) changed z in place with no warning at all on 2.5.3, although the 2.5 release notes deprecate resizing in place, so do not wait for a warning to find such calls. Their replacement, np.resize, returns a new array, and when it grows it repeats the values, as [[0, 1, 2], [3, 0, 1]] shows, where the in-place method fills the new places with zeros: its reference points out the difference, so check code that relied on the zeros. np.row_stack is gone, and newshape= raises a TypeError.

Practise: follow values through reshape and split

Two functions in plain Python compute what reshape and the split functions decide: the flat position of any index in C or F order and back, and where array_split and split cut. The sample tests run in the page:

def ravel(index, shape, order="C"):
    """Return the flat position of index in an array of this shape, in C or F order."""
    return sum(index)


def unravel(k, shape, order="C"):
    """Return the index tuple of flat position k in an array of this shape, in C or F order."""
    return tuple(k % n for n in shape)

from positions import ravel, unravel


def test_the_lessons_months():
    """finds February of year 2 and the last month in a (3, 12) table and in (3, 4, 3)"""
    assert unravel(13, (3, 12)) == (1, 1)
    assert unravel(35, (3, 4, 3)) == (2, 3, 2)
    assert ravel((2, 3, 2), (3, 4, 3)) == 35


def test_f_order():
    """lets the first index change fastest in F order"""
    assert unravel(3, (2, 3), "F") == (1, 1)
    assert unravel(7, (3, 4), "F") == (1, 2)
    assert ravel((1, 2), (3, 4), "F") == 7
    assert ravel((1, 2), (3, 4)) == 6


def test_round_trips():
    """gets back every position of a (2, 3, 4) array in both orders"""
    for order in ("C", "F"):
        assert [ravel(unravel(k, (2, 3, 4), order), (2, 3, 4), order) for k in range(24)] == list(range(24))


def test_out_of_range():
    """refuses positions and indexes that are not in the array"""
    for call in (lambda: unravel(12, (3, 4)), lambda: unravel(-1, (3, 4)), lambda: ravel((3, 0), (3, 4))):
        try:
            call()
        except ValueError:
            continue
        assert False, "an out-of-range position or index must raise ValueError"

def sections_bounds(n, sections):
    """Return the (start, stop) of each part that np.array_split(a, sections) makes from n values."""
    size = n // sections
    return [(i * size, (i + 1) * size) for i in range(sections)]


def positions_bounds(n, positions):
    """Return the (start, stop) of each part that np.split(a, positions) makes from n values."""
    return [(0, n)]

from bounds import positions_bounds, sections_bounds


def test_the_lessons_splits():
    """cuts where the lesson's run cut"""
    assert sections_bounds(7, 3) == [(0, 3), (3, 5), (5, 7)]
    assert positions_bounds(10, [3, 7]) == [(0, 3), (3, 7), (7, 10)]


def test_larger_parts_come_first():
    """gives the n % sections larger parts first, and equal parts when the sections divide n"""
    assert sections_bounds(10, 4) == [(0, 3), (3, 6), (6, 8), (8, 10)]
    assert sections_bounds(9, 3) == [(0, 3), (3, 6), (6, 9)]
    assert sections_bounds(2, 3) == [(0, 1), (1, 2), (2, 2)]


def test_positions_at_the_edges():
    """leaves an empty part for a position past the end, and one part for no positions"""
    assert positions_bounds(5, [2, 8]) == [(0, 2), (2, 5), (5, 5)]
    assert positions_bounds(5, []) == [(0, 5)]
    assert positions_bounds(5, [0, 5]) == [(0, 0), (0, 5), (5, 5)]


def test_no_sections_is_refused():
    """refuses fewer than one section"""
    try:
        sections_bounds(5, 0)
    except ValueError:
        return
    assert False, "sections = 0 must raise ValueError"

Every idea of the lesson comes back in the self-check, starting from the one your answers so far have found hardest:

Interview questions

Warm-up (fresher): What is the difference between np.concatenate and np.stack?

np.concatenate joins arrays along an axis they already have, so the result has the same number of axes and that axis gets longer: two vectors of shape (3,) become one of shape (6,). np.stack adds a new axis and puts the arrays side by side along it, so the result has one more axis: the same two vectors become (2, 3), or (3, 2) with axis=1. stack needs inputs of exactly the same shape, while concatenate needs the same number of axes and matching lengths everywhere except the joined axis. Both copy the data into a new array. vstack, hstack and column_stack are shortcuts that turn 1-D inputs into rows or columns first.

Two interview questions take the lesson to real code, a batch of images reshaped instead of transposed and a table grown row by row; the module’s question page lists the first steps and follow-ups:

Key takeaways

• reshape reads the values in C order (last index fastest) or F order (first index fastest) and lays them into the new shape in the same order; one length may be -1, and a reshape that strides can describe is a view.
• moveaxis, transpose and swapaxes reorder axes by changing strides only, expand_dims adds an axis of length 1, and squeeze with a named axis removes exactly that one.
• concatenate makes an existing axis longer and stack adds a new one; vstack, hstack, column_stack and np.block are shortcuts, and every join copies.
• split cuts into equal parts or at given positions, array_split also into unequal ones, and every part is a view of the original.
• tile and repeat write every repeat into new memory, while broadcast_to gives a read-only view with stride 0.
• On NumPy 2.5, write np.reshape, arr.view(dtype), np.vstack and np.resize instead of assigning to shape or dtype, calling np.row_stack or resizing in place.