L2.2 · LESSON 2 OF 6

Similarity and context

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

THE IDEA

Similar vectors, and vectors that depend on context

Embeddings are trained so that tokens used in similar ways get similar vectors, and cosine similarity measures that closeness. But the table gives each token one vector. The word "bank" looks the same in "river bank" and "money bank". The layers after the embedding fix that: they let every position mix in information from its neighbours, so the representation of the same token differs from sentence to sentence.

sentence:

static bank = [1, 0] (identical in every sentence) contextual = mean of [0, 1] and [1, 0] = [0.5, 0.5] cosine to bank in the OTHER sentence: 0

Real context layers (attention) mix neighbours with learned weights, not a plain average, but the effect is the same in spirit: one token, different vectors in different sentences.

A deliberately simple context layer: each token becomes the average of itself and its neighbours.

Predict first. Must the word bank have the same final hidden state in every sentence?

Chapter context

An embedding table turns a token ID into a learned vector. Sequence position must also affect the computation, because a bag of token vectors loses order. Different position mechanisms insert that information in different places.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out the two contextual versions of bank by hand, then code cosine similarity, nearest-neighbour search and the context layer.

sentence:

static bank = [1, 0] (identical in every sentence) contextual = mean of [0, 1] and [1, 0] = [0.5, 0.5] cosine to bank in the OTHER sentence: 0

Real context layers (attention) mix neighbours with learned weights, not a plain average, but the effect is the same in spirit: one token, different vectors in different sentences.

bank in "river bank":in "money bank":

cosine of the two contextual banks cosine of the static banks

To do: Static vectors: river = [0, 1], bank = [1, 0]. One context layer replaces each token by the average of itself and its neighbours. What is bank's new vector in "river bank"?
To do: Now "money bank" with money = [0, −1]. What is bank's vector?
To do: Cosine similarity between the two contextual versions of bank
To do: Cosine similarity between bank's STATIC vectors in the two sentences

Part 2. Do it in code

cosine(a, b): dot / (|a||b|); ValueError for a zero vector.
most_similar(query, E, names, k=1, exclude=None): the k names with highest cosine to query, best first (skip exclude).
contextualize(X, window=1): output row i is the mean of rows max(0, i-window) .. min(n-1, i+window) of X.
same_word_in_two_contexts(sent_a, idx_a, sent_b, idx_b): sent_a and sent_b are arrays of static vectors. Contextualize both (window 1) and return the cosine of the two chosen positions.

import numpy as np

def cosine(a, b):
    pass

def most_similar(query, E, names, k=1, exclude=None):
    pass

def contextualize(X, window=1):
    pass

def same_word_in_two_contexts(sent_a, idx_a, sent_b, idx_b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must the word bank have the same final hidden state in every sentence?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

An embedding table is $E\in\mathbb R^{V\times d}$ and token i maps to row $E_i$ .

R(\phi)=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix},\quad R(\phi)^TR(\phi)=I.

For rotary coordinates, $(R_mq)^T(R_nk)=q^TR_{n-m}k$ under a consistent rotation convention.

Basic RoPE assigns different rotation frequencies to coordinate pairs, often written $\phi_{m,j}=m b^{-2j/d}$ for position m and pair index j. The base b and any scaling or extrapolation rule are architecture settings, not universal tokenizer properties.

Worked example

A four-token vocabulary with dimension three has 4×3=12 parameters. Separately, rotating vector [1,0] by pi/2 gives [0,1] up to floating point precision. Its length remains one. Applying the same rotation to two vectors preserves their alignment; different position rotations change their relative alignment.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sin,cos,pi,isclose
x=[1.,0.]; angle=pi/2
rotated=[cos(angle)*x[0]-sin(angle)*x[1],sin(angle)*x[0]+cos(angle)*x[1]]
assert isclose(rotated[0],0,abs_tol=1e-12)
assert isclose(rotated[1],1)
assert isclose(sum(v*v for v in rotated),1)
assert 4*3 == 12

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding dimensions are not guaranteed named semantic axes. RoPE is not the same as learned absolute position embeddings. Longer supported context does not guarantee equally reliable use of every position.