L2.1 · LESSON 1 OF 6

Embedding lookup

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

From an ID to a vector

After tokenization a model has a list of integer IDs. The first layer turns each ID into a vector by lookup: the layer owns a table with one row per vocabulary entry, and token ID i simply selects row i. The table is trainable, so rows are adjusted by gradient descent like any other weights.

token id

id	dim 0	dim 1	dim 2
0	1	0	0
1	0.5	0.5	0
2	0	1	2
3	0	0	1

lookup: table[2] = [0, 1, 2] one-hot: [0, 0, 1, 0] @ table = [0, 1, 2] (same result, but lookup never builds the zeros) parameters: 4 rows × 3 columns = 12

A table with 4 tokens and 3 dimensions. Pick an ID and compare the lookup with the one-hot multiplication.

Looking up row i is the same as multiplying a one-hot vector by the table, but the lookup skips building all those zeros. It also means that in training only the rows that were actually used receive a gradient (and a token used twice receives the sum).

Predict first. A vocabulary of four tokens uses dimension three. How many table parameters?

NumPy

x = table[ids]                      # (len(ids), d)  lookup
x2 = one_hot(ids, V) @ table        # same numbers, more work

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

Count table parameters and look up a row, then write the lookup, its matrix-multiply twin and the gradient that only touches used rows.

token id

id	dim 0	dim 1	dim 2
0	1	0	0
1	0.5	0.5	0
2	0	1	2
3	0	0	1

lookup: table[2] = [0, 1, 2] one-hot: [0, 0, 1, 0] @ table = [0, 1, 2] (same result, but lookup never builds the zeros) parameters: 4 rows × 3 columns = 12

parameters (4 × 3)

row of token id 2:

parameters for 50,000 × 768

To do: A vocabulary of 4 tokens with dimension 3. How many parameters in the table?
To do: Look up token id 2 (ids start at 0) in the table shown. Type its three numbers
To do: A real model: 50,000 tokens × 768 dimensions. How many parameters in the embedding table?

Part 2. Do it in code

embed(table, ids): the rows of table for the integer ids, shape (len(ids), d). Raise ValueError for an id outside 0..V-1.
one_hot(ids, V): an array (len(ids), V) with a 1 in column id of each row.
embed_via_matmul(table, ids): the same result computed as one_hot(ids, V) @ table.
param_count(V, d): V * d.
embedding_grad(V, ids, upstream): the gradient of the loss w.r.t. the table. upstream has shape (len(ids), d). Add each upstream row into the row of its id (repeated ids add up). Return shape (V, d).

import numpy as np

def embed(table, ids):
    pass

def one_hot(ids, V):
    pass

def embed_via_matmul(table, ids):
    pass

def param_count(V, d):
    pass

def embedding_grad(V, ids, upstream):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

A vocabulary of four tokens uses dimension three. How many table parameters?

AThree: one number per dimension.BSeven: 4 + 3 numbers.CTwelve: 4 rows × 3 columns = 12 trainable numbers.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.