L2.5 · LESSON 5 OF 6

Rotary position encoding

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

Rotary position encoding (RoPE)

RoPE encodes position by rotation. Before the attention score is computed, each query and key is split into coordinate pairs, and every pair is rotated by an angle proportional to the token's position (different pairs rotate at different speeds). Because the score is a dot product of two rotated vectors, it depends on the difference of the two positions, so attention can use relative distance without any per-position table.

query position m3key position n1

each position turns the vector by 30° × position angle(q) = 90°, angle(k) = 30° → difference 60° score = q · k = cos(60°) = 0.5

offset m − n	0	1	2	3	4	6
score	1	0.866	0.5	0	-0.5	-1

Only the offset matters: (3, 1) and (10, 8) have the same score. RoPE rotates query and key coordinates in the attention step; it does not change the token IDs or add a vector to the embeddings. Real models use many coordinate pairs at different rotation speeds.

One coordinate pair, rotating 30° per position. Move the query and key positions; only m − n matters.

Predict first. Does standard RoPE rotate the vocabulary IDs themselves?

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute scores for different position pairs, find positions that give a zero score, then implement RoPE for vectors of any even size.

query position m0key position n0

each position turns the vector by 30° × position angle(q) = 0°, angle(k) = 0° → difference 0° score = q · k = cos(0°) = 1

offset m − n	0	1	2	3	4	6
score	1	0.866	0.5	0	-0.5	-1

score for m = 3, n = 1 score for m = 10, n = 8 The sliders above are for the third task: make the score 0.

To do: Query and key both start as [1, 0]. A query at position 3 is rotated by 3·30°, a key at position 1 by 1·30°. Their dot product is cos of the angle difference. Type it
To do: Now the query is at position 10 and the key at position 8. What is the dot product?
To do: Choose a query position m and key position n (0 to 12) so that the dot product is exactly 0

Part 2. Do it in code

rope_thetas(d, base=10000): the d/2 frequencies base ** (−2i/d), i = 0 .. d/2−1. d must be even (ValueError).
rope(x, pos, thetas): rotate each coordinate pair (x[2i], x[2i+1]) of the vector x by the angle pos * thetas[i] (counterclockwise). Return a new vector.
rope_score(q, k, m, n, thetas): the dot product of rope(q, m) and rope(k, n).
relative_only(q, k, thetas, offset, trials=5): check numerically that rope_score(q, k, m + offset, m) is the same for `trials` different values of m (m = 0, 3, 7, ...). Return True if all values agree within 1e-9.

import numpy as np

def rope_thetas(d, base=10000):
    pass

def rope(x, pos, thetas):
    pass

def rope_score(q, k, m, n, thetas):
    pass

def relative_only(q, k, thetas, offset, trials=5):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does standard RoPE rotate the vocabulary IDs themselves?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.