ML7.5 · LESSON 5 OF 6

PCA and visual embeddings

Perform a clustering update and explain dimensionality reduction, mixtures and anomaly detection.

THE IDEA

Finding the direction where the data varies most

High-dimensional data is hard to see and often redundant: columns that rise and fall together carry the same information twice. Principal component analysis (PCA) finds a new set of perpendicular axes, ordered by how much of the data's spread each one captures. The first axis points along the direction of largest variance; the second is perpendicular to it with the next most, and so on.

Below is a cloud of 24 points. Turn your own axis and watch the variance of the points projected onto it (the green lines are the projections). At one angle it is largest: that is the first principal component. The leftover distance from the points to the axis is smallest at the same angle, because projected variance plus leftover always equals the total.

your axis = 0°keep k = 0

first feature in 5x smaller units (x multiplied by 5) show the principal axis (orange)

datayour axis and its projection linesreconstruction from k components (hollow)

mean = (6.05, 4.61) covariance (divide by n - 1 = 23): [ 4.35 2.68 ] [ 2.68 1.96 ] trace = 4.35 + 1.96 = 6.3 det = 4.35 x 1.96 - 2.68^2 = 1.32 lambda = 3.15 +/- sqrt(3.15^2 - 1.32) = 3.15 +/- 2.94 lambda1 = 6.09 lambda2 = 0.22 v1 = (b, lambda1 - a) = (2.68, 1.74), length-1: (0.84, 0.54) angle of v1: 33 degrees

variance along your axis (0 deg): 4.35 (69% of 6.3) largest possible (lambda1): 6.09 (96.6%) keeping k = 0: mean squared reconstruction error 6.04

Turn the axis: 1.74 of variance is still unused. Lines to your axis shorten as it improves.

Turn the axis. Tick 'show the principal axis' to check yourself after you have tried.

The arithmetic: an eigen-decomposition

Centre the data (subtract the mean), then compute the covariance matrix. Here it is [[4.35, 2.68], [2.68, 1.96]]: the variances of x and y on the diagonal, and how they move together (the covariance 2.68) off it. The principal axes are its eigenvectors, and the variance along each is its eigenvalue. For a 2 × 2 matrix the eigenvalues satisfy: sum = trace = 6.3 and product = determinant = 1.32. Solving gives λ₁ = 6.09 and λ₂ = 0.22, so the first component explains 96.6% of the total variance and the second 3.4%. The first eigenvector is (0.84, 0.54), pointing 33° above the x axis. In general you let a library do this, for example np.linalg.eigh on the covariance matrix.

Keeping fewer components: reconstruction

Project each point onto the first k axes and map it back: you get the best k-dimensional approximation of the data. With k = 2 you recover the data exactly; with k = 0 every point collapses to the mean (mean squared error 6.04); with k = 1 the points lie on a line and the mean squared error is 0.21, which is λ₂ times (n − 1) / n. So the error of dropping components is the variance you dropped. Pick k by how much variance you are willing to lose: here one component keeps 96.6%.

Predict first. You measure the first feature in units five times smaller (multiply it by 5) and rerun PCA. What happens to the first principal axis?

Visual embeddings: t-SNE and UMAP are not PCA

PCA is a linear rotation plus truncation, so distances in the plot are distances you could compute from the kept components. t-SNE and UMAP are nonlinear methods that try to keep each point's neighbours as neighbours when squeezing into two dimensions. They are good for seeing local groups, and that is all they promise. The size of a cluster, the empty space between two clusters, and the relative position of clusters can depend on settings and on the random seed, and should not be read as distances in the original space. A large gap in the picture does not prove the groups are far apart. Use embeddings to form hypotheses; check them in the original features.

Chapter context

Unsupervised methods expose structure according to their objectives and assumptions. They do not discover a single objectively correct organization of a dataset; changing distance, dimension or density assumptions can change the result.

TRY IT YOURSELF

Part 1. Do it on the plot

The panel shows the covariance, its eigenvalues and the variance along your axis, recomputed live. Variance along your axis at 0 degrees is 4.35; the maximum is 6.09.

your axis = 0°keep k = 0

first feature in 5x smaller units (x multiplied by 5) show the principal axis (orange)

datayour axis and its projection linesreconstruction from k components (hollow)

variance along your axis (0 deg): 4.35 (69% of 6.3) largest possible (lambda1): 6.09 (96.6%) keeping k = 0: mean squared reconstruction error 6.04

Turn the axis: 1.74 of variance is still unused. Lines to your axis shorten as it improves.

To do: Turn your own axis until it lies within 1 degree of the direction of largest variance (with the original units)
To do: Choose the smallest number of components that keeps at least 95 percent of the variance
To do: Measure the first feature in 5 times smaller units (tick the box), then line your axis up with the new principal axis within 1 degree

Part 2. Do it in code

pca(X) centres X (n rows, d columns), forms the covariance matrix with n - 1 in the denominator, and uses np.linalg.eigh. It returns (components, ratio, mean): components has one unit eigenvector per ROW, ordered from largest to smallest eigenvalue; ratio is each eigenvalue divided by the total (all zeros if the total variance is 0); mean is the column mean. Raise ValueError if n < 2.
Sign convention: eigenvectors are only defined up to sign, so flip each one so its component of largest absolute value is positive (first one wins a tie).
reconstruct(X, k) keeps only the first k components: project the centred data, map back and add the mean. k = 0 gives the mean for every row; k = d gives X back.

import numpy as np

def pca(X):
    # return (components, ratio, mean); eigh returns eigenvalues in ASCENDING order
    pass

def reconstruct(X, k):
    # rebuild X from the first k principal components
    pass

X = np.array([[1.0, 2.0], [2.0, 4.1], [3.0, 5.9], [4.0, 8.2]])
print(pca(X))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a large gap in a t-SNE picture prove a large original-space distance?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

J=\sum_i\|x_i-\mu_{c_i}\|^2,\quad \mu_k=\frac1{|C_k|}\sum_{i\in C_k}x_i.

For centered data X, PCA uses leading eigenvectors of $X^TX/(n-1)$ , or an equivalent SVD. Centering and feature scaling are separate choices.

For a Gaussian mixture, EM responsibilities are

r_{ik}=\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_j\pi_j\mathcal N(x_i\mid\mu_j,\Sigma_j)}.

Mixing weights pi are nonnegative and sum to one; each row of responsibilities also sums to one. Weighted sufficient-statistic updates differ from hard k-means assignments.

Worked example

With points [0,2,8,10] and starting centers [0,10], assignments are [0,2] and [8,10]. Updated means are 1 and 9. Squared errors are 1,1,1,1, totaling 4. This example gives one iteration; it does not prove that every initialization finds the best partition.

Reference code

The following example uses the Python standard library and does not call a remote service.

points, centers = [0,2,8,10], [0,10]
groups = [[x for x in points if min(range(2), key=lambda j: (x-centers[j])**2) == i] for i in range(2)]
updated = [sum(group)/len(group) for group in groups]
assert updated == [1,9]
assert sum((x-c)**2 for group,c in zip(groups,updated) for x in group) == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clusters are objective-dependent. PCA maximizes variance, which may not correspond to useful semantic structure. Anomaly detection is not the same as identifying known fraud classes.