F3.4 · LESSON 4 OF 6

Reductions and vectorization

Use array shapes, broadcasting and reductions to express a small data transformation.

THE IDEA

A reduction collapses an axis

A reduction combines many numbers into one: sum, mean, max. On a table you choose which axis disappears. With axis=0 the rows are collapsed, so you get one answer per column. With axis=1 the columns are collapsed, so you get one answer per row. With no axis, everything collapses to a single number.

operation axis keepdims

M, shape (3, 4); same colour = combined together

result, shape (4,)

M.sum(axis=0) -> [10, 11, 16, 9] shape (4,)

The axis you name is the one that disappears. axis=0 walks down each column and leaves one number per column. axis=1 walks along each row and leaves one number per row.

Same colour means combined together. The axis you name is the one that disappears from the shape. keepdims leaves it behind with length 1, which is handy for broadcasting back.

Predict first. X has shape (100, 5): 100 examples, 5 features. Which call gives the mean of each feature?

Vectorization: let compiled loops do the stepping

A Python for loop asks the interpreter to run a few instructions for every element. An expression like a * x + b on arrays hands the whole job to compiled code: one call per operation, with the element loop inside. The arithmetic is the same; what disappears is the per-element interpreter overhead. We will count that instead of timing it, because timings change from machine to machine.

n = 10^3 numbers

Compute y = a * x + b for every element of x.

Python loopfor i in range(n): y[i] = a * x[i] + b1,000 trips through the interpreter loopVectorizedy = a * x + b2 array operations, each runs its inner loop in compiled code

Arithmetic done is the same either way: 2,000 multiplications and additions. What changes is who steps through the elements: 1,000 interpreter iterations versus 2 calls. The price: a * x creates 1 temporary array of 1,000 float64 values = 0.01 MB (8 bytes each), plus the result array.

Drag n. The count of interpreter iterations grows with n; the number of array operations does not. The temporary array is the price.

The common mistake: an overflow that vectorization makes fast and wrong

softmax turns scores into probabilities. The naive version breaks on large scores

>>> scores = np.array([1000., 1000.])
>>> e = np.exp(scores)
>>> e / e.sum()
array([nan, nan])        # exp(1000) overflows to inf, and inf / inf is nan

>>> e = np.exp(scores - scores.max())   # subtract the max first: same answer mathematically
>>> e / e.sum()
array([0.5, 0.5])

Subtracting the row maximum does not change softmax (it cancels in the ratio) but keeps every exponent at or below 0. For a table, keepdims=True keeps the maximum as a column so it broadcasts back across each row.

Chapter context

A tensor is a multidimensional array with a shape and element type. Many AI bugs come from doing a valid operation along the wrong axis. Write the intended dimensions before trying to shorten the code.

TRY IT YOURSELF

row_means(X) returns one mean per row, col_max(X) one maximum per column.
softmax_rows(X) turns each row into probabilities that sum to 1. Subtract the row maximum first so large numbers do not overflow.
loop_scores is given and slow. Write scores(X, w) with the same result and no Python loops, comprehensions or while loops.

import numpy as np

def row_means(X):
    # one mean per row -> shape (n,)
    pass

def col_max(X):
    # one maximum per column -> shape (d,)
    pass

def softmax_rows(X):
    # exp(x - row max) / row sum of that, row by row
    pass

def loop_scores(X, w):
    out = []
    for i in range(X.shape[0]):
        total = 0.0
        for j in range(X.shape[1]):
            total += X[i, j] * w[j]
        out.append(total)
    return np.array(out)

def scores(X, w):
    # same numbers as loop_scores, vectorized: no for, no while, no comprehension
    pass

M = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
print(row_means(M), col_max(M))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Which reduction finds each row's mean in a 2D array?

Amean(axis=0), because axis 0 runs across the columns of each row.Bmean(), because with no axis NumPy averages each row separately.Cmean(axis=1), because axis 1 runs across the columns.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a data matrix $X\in\mathbb R^{n\times d}$ , feature means are

\mu_j=\frac1n\sum_{i=1}^n X_{ij},\qquad Z_{ij}=X_{ij}-\mu_j.

Here i indexes examples and j indexes features. The mean vector has d entries, so subtracting it from X broadcasts along rows.

Worked example

For X = [[1, 2], [3, 4]], the feature means are [2, 3]. Centering produces [[-1, -1], [1, 1]]. Each centered column has mean zero. Computing row means instead would answer a different question even though the operation is valid.

Reference code

The following example uses the Python standard library and does not call a remote service.

X = [[1, 2], [3, 4]]
means = [sum(row[j] for row in X) / len(X) for j in range(2)]
centered = [[x - m for x, m in zip(row, means)] for row in X]
assert means == [2, 3]
assert centered == [[-1, -1], [1, 1]]

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Broadcasting is a shape rule, not permission to ignore axis meanings. Vectorized code can create large intermediate arrays. Normalize training and test data with training statistics only.