L2.6 · LESSON 6 OF 6

Geometry and evaluation

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

Looking at high dimensions in two

Embeddings have hundreds of dimensions, and people plot them in 2-D. Any such picture is a projection, and projecting throws information away. Points that are far apart can land on top of each other, and clusters or directions you see may depend on the projection method you chose.

drop the coordinate:

pair	true distance (3-D)	distance in the plot
A–B	1	1
A–C	10	0
B–C	10.05	1

A, B, C are fixed: A = (0,0,0), B = (1,0,0), C = (0,0,10). Dropping a coordinate is the simplest projection; methods like PCA choose the directions that keep the most spread, but any 2-D picture of high-dimensional data distorts some neighbour relations.

Three points that are clearly different in 3-D. Choose which coordinate to drop and watch the distances.

Predict first. Does a 2-D plot preserve every high-dimensional neighbour relation?

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

See how a projection distorts distances and neighbours, then write distance, projection, neighbour-agreement and PCA code.

drop the coordinate:

pair	true distance (3-D)	distance in the plot
A–B	1	1
A–C	10	0
B–C	10.05	1

true distance A–C A–C after dropping z

in that plot A's nearest neighbour (B or C) coordinate to drop that keeps A–C ≥ 5 (x, y or z)

To do: Three points in 3-D: A = (0,0,0), B = (1,0,0), C = (0,0,10). What is the true distance between A and C?
To do: Project to 2-D by dropping the z coordinate. What is the distance between A and C in the plot?
To do: In that plot, which point looks like A's nearest neighbour: B or C? Type the letter
To do: Choose which coordinate to drop (x, y or z) so that A and C stay at least 5 apart in the plot

Part 2. Do it in code

pairwise_distances(X): the (n, n) matrix of Euclidean distances between the rows of X.
project(X, keep): keep only the listed columns of X.
nn_agreement(X, Y): for each row, find its nearest other row in X and in Y (rows correspond). Return the fraction of rows where the nearest neighbour is the same.
pca_project(X, k): centre X, take the top-k right singular vectors, and return the centred data projected onto them, shape (n, k).

import numpy as np

def pairwise_distances(X):
    pass

def project(X, keep):
    pass

def nn_agreement(X, Y):
    pass

def pca_project(X, k):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a 2D plot preserve every high-dimensional neighbor relation?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.