F3.3 · LESSON 3 OF 6

Broadcasting

Use array shapes, broadcasting and reductions to express a small data transformation.

THE IDEA

Combining arrays of different shapes

Adding a table to a vector looks wrong at first: how do 6 numbers and 3 numbers add up? Broadcasting is the rule that answers it. NumPy compares the two shapes from the right. Two lengths are compatible if they are equal or one of them is 1. A missing leading axis counts as 1. A length 1 is stretched (conceptually repeated) to match the other.

A row of 3 numbers is applied to every row of the table

>>> X = np.array([[1, 2, 3], [4, 5, 6]])     # shape (2, 3)
>>> b = np.array([10, 20, 30])                # shape (3,)
>>> X + b
array([[11, 22, 33],
       [14, 25, 36]])

Aaxes length 0 length 1

Baxes length 0

axis (from the right)	A	B	verdict
last	3	3	equal: fine
last - 1	2	·	B has no axis: treated as 1

A.shape (2, 3) + B.shape (3,) -> result shape (2, 3)

Step through the result cells

A stretched (dashed = copied)

B stretched (dashed = copied)

A + B

Cell [0, 0]: 1 + 10 = 11. A is filled with 1, 2, 3, ... and B with 10, 20, 30, ... NumPy does not really build the dashed copies; it reuses the same values, which is why broadcasting is cheap.

Choose two shapes. The table compares the axes from the right; the grids stretch the smaller array across the larger one; drag the slider to follow one result cell at a time.

Check yourself first

Predict first. What is the shape of the result of np.ones((3, 1)) + np.ones((1, 4))?

Predict first. Can (2, 3) and (2,) be added?

The common mistake: right shape rule, wrong axis meaning

Centering a table means subtracting each column mean. If you reduce along the wrong axis you either get an error or, worse, a quietly wrong answer on a square table:

Row means have shape (2,), but we need something that lines up with the columns

>>> X = np.array([[1., 2., 3.], [4., 5., 6.]])
>>> X - X.mean(axis=1)
ValueError: operands could not be broadcast together with shapes (2,3) (2,)

>>> X - X.mean(axis=1, keepdims=True)    # shape (2,1): subtracts each ROW mean
array([[-1.,  0.,  1.],
       [-1.,  0.,  1.]])

>>> X - X.mean(axis=0)                   # shape (3,): subtracts each COLUMN mean
array([[-1.5, -1.5, -1.5],
       [ 1.5,  1.5,  1.5]])

Chapter context

A tensor is a multidimensional array with a shape and element type. Many AI bugs come from doing a valid operation along the wrong axis. Write the intended dimensions before trying to shorten the code.

TRY IT YOURSELF

can_broadcast(s1, s2) takes two shape tuples and says whether NumPy could combine them. Compare from the right: equal or one of them 1, missing leading axes count as 1.
center_columns(X) subtracts each column mean. standardize(X) also divides by the column standard deviation, but a constant column becomes zeros, never NaN.
pairwise_diff(v) returns the (n, n) table with entry [i, j] = v[i] - v[j], using broadcasting and no loop.

import numpy as np

def can_broadcast(s1, s2):
    # shapes are tuples of ints, for example (2, 3) and (3,)
    pass

def center_columns(X):
    # subtract each column's mean
    pass

def standardize(X):
    # (X - column mean) / column std, but a column with std 0 becomes all zeros
    pass

def pairwise_diff(v):
    # result[i, j] = v[i] - v[j]
    pass

print(can_broadcast((2, 3), (3,)))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can shapes (2, 3) and (3,) be added?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a data matrix $X\in\mathbb R^{n\times d}$ , feature means are

\mu_j=\frac1n\sum_{i=1}^n X_{ij},\qquad Z_{ij}=X_{ij}-\mu_j.

Here i indexes examples and j indexes features. The mean vector has d entries, so subtracting it from X broadcasts along rows.

Worked example

For X = [[1, 2], [3, 4]], the feature means are [2, 3]. Centering produces [[-1, -1], [1, 1]]. Each centered column has mean zero. Computing row means instead would answer a different question even though the operation is valid.

Reference code

The following example uses the Python standard library and does not call a remote service.

X = [[1, 2], [3, 4]]
means = [sum(row[j] for row in X) / len(X) for j in range(2)]
centered = [[x - m for x, m in zip(row, means)] for row in X]
assert means == [2, 3]
assert centered == [[-1, -1], [1, 1]]

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Broadcasting is a shape rule, not permission to ignore axis meanings. Vectorized code can create large intermediate arrays. Normalize training and test data with training statistics only.