M3.3 · LESSON 3 OF 6

Transpose and orientation

Multiply matrices, track dimensions and interpret transformations, inverses, eigenvectors and SVD.

THE IDEA

Swapping rows and columns

The transpose Aᵀ of a matrix writes each row as a column: entry (i, j) of A becomes entry (j, i) of Aᵀ. So a matrix of shape (m, n) becomes shape (n, m). A (2, 5) matrix becomes (5, 2). The transpose is how you turn a row into a column and back, and it is the glue that makes shapes line up.

rows columns shape of A: (2, 3) and of A transposed: (3, 2)

A =Aᵀ =142536

The highlighted entry sits at row 1, column 3 of A (value 3) and at row 3, column 1 of Aᵀ. A column of A becomes a row of Aᵀ.

Choose a shape and move over the entries of A. The matching entry in Aᵀ lights up.

Predict first. A matrix has shape (2, 5). What is the shape of its transpose?

The product flips order

(AB)ᵀ = BᵀAᵀ. The order reverses, exactly like taking off shoes and socks: whatever you did last you undo first. Shapes insist on it: if A is (m, n) and B is (n, p), then (AB)ᵀ is (p, m), and only BᵀAᵀ, which is (p, n) times (n, m), has that shape.

A = [[1, 2], [3, 4]], B = [[2, 0], [1, 2]]

AB = [[4, 4], [10, 8]]

(AB)ᵀ = [[4, 10], [4, 8]]

BᵀAᵀ = [[4, 10], [4, 8]] same

AᵀBᵀ = [[2, 7], [4, 10]] different

Computed, not claimed. The tempting AᵀBᵀ gives a different matrix.

Shape tracking in a neural-network layer

A layer turns a batch of inputs into a batch of outputs. X holds one example per row, shape (batch, inputs). In PyTorch a Linear layer stores its weight W with shape (outputs, inputs), and computes X @ W.T, giving (batch, outputs). The transpose is not decoration: it moves the matching "inputs" size to the inside of the product. Set up the layer below and compare the five possible expressions.

batch size inputs per example outputs

expression

X: (5, 3)

W: (4, 3) stored

W.T: (3, 4)

X.T: (3, 5)

X @ W.T: (5, 3) times (3, 4)

Runs. Result shape (5, 4). That is (batch, outputs): the right shape.

Try 32 examples of 784 numbers with 10 outputs. Only X @ W.T gives (32, 10). W @ X.T runs too, but returns the transposed layout (10, 32).

Predict first. X has shape (32, 784) and W is stored as (10, 784). What shape does X @ W.T have?

The bug that shapes cannot catch

When a layer has as many outputs as inputs (a square weight matrix), X @ W and X @ W.T both run. Only one of them is the layer you meant. Shape errors protect you only when the shapes differ.

Both run, only one is right

import numpy as np
X = np.array([[1.0, 0.0]])
W = np.array([[1.0, 2.0],
              [3.0, 4.0]])      # stored (out, in)
X @ W.T     # [[1., 3.]]  the intended output
X @ W       # [[1., 2.]]  runs silently, wrong meaning

Chapter context

A matrix represents a linear map once input and output bases are chosen. Multiplication composes maps. Tracking shapes is an algebraic requirement and also a practical way to detect implementation mistakes before running code.

TRY IT YOURSELF

Part 1. Do it by hand

The weight W is stored as (outputs, inputs). Change the sizes and the expression and read the result or the error.

batch size inputs per example outputs

expression

X: (5, 3)

W: (4, 3) stored

W.T: (3, 4)

X.T: (3, 5)

X @ W.T: (5, 3) times (3, 4)

Runs. Result shape (5, 4). That is (batch, outputs): the right shape.

To do: Make the layer output shape (32, 10) with a valid expression
To do: Pick the wrong expression X @ W and make it fail with a shape error
To do: Find the dangerous case: X @ W runs without any error even though W is stored (out, in)

Part 2. Do it in code

Write my_transpose(A) with loops (no .T): entry (i, j) of the result is entry (j, i) of A.
Write dense(X, W, b): the layer output X @ W.T + b, where W is stored as (out, in) like a PyTorch Linear weight. Raise ValueError with both shapes in the message when X has the wrong number of features, or b the wrong length.
Write rule_holds(A, B): True when (AB) transposed equals B transposed times A transposed (use np.allclose).

import numpy as np

def my_transpose(A):
    # swap rows and columns using loops
    pass

def dense(X, W, b):
    # X is (batch, in), W is (out, in), b is (out,)  ->  (batch, out)
    pass

def rule_holds(A, B):
    # is (A B)^T equal to B^T A^T ?
    pass

print(my_transpose([[1, 2, 3], [4, 5, 6]]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is the transpose shape of a (2,5) matrix?

A(2,5), because transposing only changes the entries, not the shape.B(10,1), because transposing flattens the matrix into a column.C(5,2).

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

C_{ij}=\sum_k A_{ik}B_{kj},\quad (AB)^T=B^TA^T.

\det\begin{bmatrix}a&b\\c&d\end{bmatrix}=ad-bc,\quad Av=\lambda v.

A=U\Sigma V^T.

U and V contain orthonormal singular directions; diagonal entries of Sigma are singular values. Eigenvalues and singular values are different concepts.

For an invertible 2x2 matrix, the inverse is

\begin{bmatrix}a&b\\c&d\end{bmatrix}^{-1}=\frac1{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}.

This formula requires a nonzero determinant. A near-zero determinant can signal numerical sensitivity, so exact invertibility alone does not guarantee a well-conditioned computation.

Worked example

Multiply A=[[1,2],[3,4]] by B=[[2,0],[1,2]]. Row-column products give C=[[4,4],[10,8]]. The determinant of A is 1×4−2×3=−2, so A is invertible. In general AB is not equal to BA; their shapes may even differ.

Reference code

The following example uses the Python standard library and does not call a remote service.

A, B = [[1,2],[3,4]], [[2,0],[1,2]]
C = [[sum(A[i][k]*B[k][j] for k in range(2)) for j in range(2)] for i in range(2)]
assert C == [[4,4],[10,8]]
assert A[0][0]*A[1][1] - A[0][1]*A[1][0] == -2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

AB usually differs from BA. A transpose is not generally an inverse. A negative eigenvalue is possible, while singular values are nonnegative.