L2.4 · LESSON 4 OF 6

Rotations in two dimensions

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

Rotating a vector

A rotation turns a vector by an angle without changing its length. In two dimensions, rotating (x, y) counterclockwise by θ gives (x cos θ − y sin θ, x sin θ + y cos θ). Rotating [1, 0] by 90° gives [0, 1]; rotating [3, 4] by 90° gives [−4, 3], still of length 5.

The key property for position encoding: rotate two vectors by the same angle and their dot product does not change; rotate them by different angles and the dot product depends only on the difference of the angles.

rotate [3, 4] by (degrees)90

rotated = [-4, 3] length 5 (always 5)

first copy of [1, 0] rotated by30second copy rotated by70

dot = cos(70 − 30) = cos(40°) = 0.766 (depends only on the difference)

Top: rotate [3, 4]. Bottom: two copies of [1, 0] rotated by different angles.

Predict first. Rotate [1, 0] by 90 degrees counterclockwise. What do you get?

NumPy

R = np.array([[c, -s], [s, c]])    # c = cos(theta), s = sin(theta)
R @ v                              # rotated copy of v

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

Rotate vectors by hand and compute a dot product of two rotated copies. Then write the rotation matrix and test its properties in Python.

rotate [3, 4] by (degrees)90

rotated = [-4, 3] length 5 (always 5)

first copy of [1, 0] rotated by30second copy rotated by70

dot = cos(70 − 30) = cos(40°) = 0.766 (depends only on the difference)

[1, 0] rotated 90°:[3, 4] rotated 90°:

dot of the copies rotated by 30° and 70°

To do: Rotate [1, 0] by 90° counterclockwise. Type the result (round tiny errors to 0)
To do: Rotate [3, 4] by 90° counterclockwise: (x, y) → (−y, x). Type the result
To do: Two copies of [1, 0]: one rotated by 30°, the other by 70°. Their dot product equals the cosine of the angle between them (40°). Type it with 3 decimals

Part 2. Do it in code

rot(theta): the 2×2 counterclockwise rotation matrix for an angle in radians.
rotate(v, theta): rotate the vector v by theta radians.
angle_between(a, b): the angle in radians between two non-zero vectors (0 to pi), using arccos of the clipped cosine.
dot_after_rotations(a, b, ta, tb): rotate a by ta and b by tb (radians) and return their dot product.

import numpy as np

def rot(theta):
    pass

def rotate(v, theta):
    pass

def angle_between(a, b):
    pass

def dot_after_rotations(a, b, ta, tb):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Rotate [1,0] by 90 degrees counterclockwise.

A[0, −1], because counterclockwise rotation points downward.B[1, 0], because a rotation never changes a vector.C[0, 1]: x cos 90° − y sin 90° = 0 and x sin 90° + y cos 90° = 1.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.