L2.3 · LESSON 3 OF 6

Position information

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

Order has to be added on purpose

An embedding lookup is the same for a token wherever it appears, and a sum or average of embeddings forgets order entirely: "dog bites man" and "man bites dog" are the same bag of words. Attention by itself also treats its inputs as a set. To use word order the model must be told where each token sits.

Learned absolute positions: a second trainable table, one row per position, added to the token vector.
Sinusoidal positions: fixed sine and cosine waves of different frequencies give every position its own pattern.
Rotary methods, next lessons, work inside attention instead of adding a vector.

add position vectors

position	word	vector the model gets
0	dog	[1, 0]
1	bites	[1, 1]
2	man	[3, 1]

sum over positions = [5, 2] in both orders With positions, each position's vector now depends on the order (compare the table above).

sinusoidal, d = 4	dim 0	dim 1	dim 2	dim 3
position 0	0	1	0	1
position 1	0.841	0.54	0.01	1
position 2	0.909	-0.416	0.02	1
position 3	0.141	-0.99	0.03	1

Same words, two orders. Switch position vectors on and off, and look at the fixed sinusoidal table below.

Predict first. Do the same tokens in reversed order always express the same sentence?

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

Show that a bag of vectors forgets order, add positions by hand, then generate the sinusoidal table in Python.

add position vectors

position	word	vector the model gets
0	dog	[1, 0]
1	bites	[1, 1]
2	man	[3, 1]

sum over positions = [5, 2] in both orders With positions, each position's vector now depends on the order (compare the table above).

sinusoidal, d = 4	dim 0	dim 1	dim 2	dim 3
position 0	0	1	0	1
position 1	0.841	0.54	0.01	1
position 2	0.909	-0.416	0.02	1
position 3	0.141	-0.99	0.03	1

sum of dog, bites, man:third position of "man bites dog" (+p2):

PE(1, 0) PE(2, 1)

To do: dog = [1, 0], bites = [0, 1], man = [1, 1]. Add the three vectors of "dog bites man". (The sum for "man bites dog" is the same.) Type the sum
To do: Add position vectors p0 = [0,0], p1 = [1,0], p2 = [2,0] to each token. In "man bites dog", what is the vector at the THIRD position (dog + p2)?
To do: Sinusoidal position encoding, dimension 0: PE(pos) = sin(pos / 10000^0) = sin(pos). Type PE(1, 0) with 3 decimals
To do: Dimension 1 uses cosine: PE(pos, 1) = cos(pos). Type PE(2, 1) with 3 decimals

Part 2. Do it in code

bag_sum(E, ids): the sum of the embedding rows of the ids (a vector). Order does not matter.
sinusoidal(n_pos, d): the (n_pos, d) table with PE[pos, 2i] = sin(pos / 10000^(2i/d)) and PE[pos, 2i+1] = cos(pos / 10000^(2i/d)). d must be even (ValueError).
add_positions(X, P): X is (n, d); add the first n rows of P. ValueError if P has fewer than n rows.
order_sensitive(E, ids, P): True if reversing the ids changes the position-aware matrix add_positions(E[ids], P), False if it stays the same.

import numpy as np

def bag_sum(E, ids):
    pass

def sinusoidal(n_pos, d):
    pass

def add_positions(X, P):
    pass

def order_sensitive(E, ids, P):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Do the same tokens in reversed order always express the same sentence?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.