M3.1 · LESSON 1 OF 6

Matrix-vector products

Multiply matrices, track dimensions and interpret transformations, inverses, eigenvectors and SVD.

THE IDEA

A matrix is a machine that turns vectors into vectors

A matrix is a rectangular grid of numbers. Its job in this course: take a vector in, give a vector out. A matrix with m rows and n columns (shape (m, n)) accepts an n-vector and returns an m-vector. A (2, 3) matrix eats a vector with 3 coordinates and returns one with 2. A (3, 2) matrix does the opposite. Without this rule you could not tell whether a model layer takes 784 pixel values and gives 10 class scores, or the other way round.

Take A = 1234 and x = [1, 0]. Each output coordinate is the dot product of one row of A with x (you met dot products in M2.2): row 1 · x = 1·1 + 2·0 = 1, row 2 · x = 3·1 + 4·0 = 3. So A x = [1, 3].

The same product, read by columns

There is a second reading that gives the same numbers. A x is a weighted sum of the columns of A, with the entries of x as the weights: A x = x₁·(column 1) + x₂·(column 2). For x = [1, 0] that is 1·[1, 3] + 0·[2, 4] = [1, 3].

This explains the picture. The vectors i = [1, 0] and j = [0, 1] land on column 1 and column 2 of A, because A i is 1·col 1 + 0·col 2. Every other point is built from i and j, so the whole grid moves with them. A matrix is a recipe for where the two basis arrows go; everything else follows.

A = x = [2, 1]

Rows view (dot products)

out 1 = [2, 1] · [2, 1] = 2·2 + 1·1 = 5

out 2 = [0, 1] · [2, 1] = 0·2 + 1·1 = 1

Columns view (weighted sum)

2·[2, 0] + 1·[1, 1]

= [4, 0] + [1, 1] = [5, 1]

Drag the grey dot. The blue dashed arrow is x₁ copies of i', the orange one continues with x₂ copies of j', and they end exactly at the green A x. Both views always agree.

Drag the grey dot x (or edit the matrix). Left: the grid after the matrix, blue i and orange j are the columns. Right: the same product as row dot products and as a weighted sum of columns. They always agree.

Predict first. What is [[1, 2], [3, 4]] times [1, 0]?

What goes wrong: shapes, and the vectors that vanish

The shapes must agree: the number of columns of A must equal the number of entries of x. Otherwise there is nothing to multiply row against.

A shape error is a useful early warning

import numpy as np
A = np.array([[1, 2, 3],
              [4, 5, 6]])      # shape (2, 3): takes a 3-vector
x = np.array([1, 1])           # only 2 entries
A @ x
# ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0 ...

A @ np.array([1, 1, 1])        # array([ 6, 15])  -> shape (2,)

Chapter context

A matrix represents a linear map once input and output bases are chosen. Multiplication composes maps. Tracking shapes is an algebraic requirement and also a practical way to detect implementation mistakes before running code.

TRY IT YOURSELF

Part 1. Do it by hand

The grid, the unit square and a purple F-shaped test figure move with your matrix. Tasks tick from the real numbers and stay ticked once earned.

A =

Type entries, or focus a box and press the up and down arrow keys. The blue arrow is where i = [1, 0] lands (column 1), the orange one is j = [0, 1] (column 2).

A · [1, 0]	[1, 0]
A · [0, 1]	[0, 1]
A · [1, 1]	[1, 1]
A · [1, 2]	[1, 2]
det A	1

To do: Rotate the plane a quarter turn counterclockwise: send i = [1, 0] to [0, 1] and j = [0, 1] to [-1, 0]
To do: Shear the plane so that every point [x, y] lands on [x + y, y] (heights stay, the top slides right)
To do: Find any matrix that sends [1, 1] to [5, 1]
To do: Find a matrix that is not all zeros but sends [1, 2] to the zero vector [0, 0]

Part 2. Do it in code

Write mat_vec_rows(A, x): each output entry is the dot product of one row of A with x. Use plain loops or sum, not @.
Write mat_vec_cols(A, x): the same answer built as a weighted sum of the columns of A, with the entries of x as the weights.
Both must raise ValueError when A has a different number of columns than x has entries. A may be non-square.

import numpy as np

def mat_vec_rows(A, x):
    # entry i is the dot product of row i of A with x
    pass

def mat_vec_cols(A, x):
    # x[0] * column 0 + x[1] * column 1 + ...
    pass

print(mat_vec_rows([[1, 2], [3, 4]], [1, 0]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is [[1,2],[3,4]] times [1,0]?

A[4,6], because the entries of each column are added together.B[1,2], because the product returns the first row of the matrix.C[1,3], the first column.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

C_{ij}=\sum_k A_{ik}B_{kj},\quad (AB)^T=B^TA^T.

\det\begin{bmatrix}a&b\\c&d\end{bmatrix}=ad-bc,\quad Av=\lambda v.

A=U\Sigma V^T.

U and V contain orthonormal singular directions; diagonal entries of Sigma are singular values. Eigenvalues and singular values are different concepts.

For an invertible 2x2 matrix, the inverse is

\begin{bmatrix}a&b\\c&d\end{bmatrix}^{-1}=\frac1{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}.

This formula requires a nonzero determinant. A near-zero determinant can signal numerical sensitivity, so exact invertibility alone does not guarantee a well-conditioned computation.

Worked example

Multiply A=[[1,2],[3,4]] by B=[[2,0],[1,2]]. Row-column products give C=[[4,4],[10,8]]. The determinant of A is 1×4−2×3=−2, so A is invertible. In general AB is not equal to BA; their shapes may even differ.

Reference code

The following example uses the Python standard library and does not call a remote service.

A, B = [[1,2],[3,4]], [[2,0],[1,2]]
C = [[sum(A[i][k]*B[k][j] for k in range(2)) for j in range(2)] for i in range(2)]
assert C == [[4,4],[10,8]]
assert A[0][0]*A[1][1] - A[0][1]*A[1][0] == -2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

AB usually differs from BA. A transpose is not generally an inverse. A negative eigenvalue is possible, while singular values are nonnegative.