M2.6 · LESSON 6 OF 6

Vectors in learning

Compute vector operations, lengths, angles and projections and explain what each measures.

THE IDEA

Where vectors show up in learning

A feature vector describes one example with one number per measured property: a house as [area in m², bedrooms, age in years] = [120,3,15], another as [90,2,40]. An embedding is a vector that a model learned to represent a word, sentence or image, usually with hundreds of coordinates. Its individual coordinates rarely mean anything you could name, but distances and angles between embeddings carry information, and that is why this chapter's operations matter: dot products, norms, cosine and projection are what models and search systems actually compute.

Nearest neighbour: cosine or distance?

A common job: find the vector most similar to a query. You can rank by cosine similarity (same direction) or by Euclidean distance (nearby position). They answer different questions, and they can disagree. Below, a toy two-dimensional vocabulary has invented coordinates. Drag the blue query. Purple ring: nearest by cosine. Orange ring: nearest by distance.

normalize to length 1

They disagree. Cosine (purple ring) picks dog; distance (orange ring) picks pet.

word	cosine	distance
dog ◯c	0.999	5.315
pet ◯d	0.894	1.414
fish	0.394	5
hug	0.196	3.041
love	0.141	5.385
bone	-0.196	3.64
car	-0.97	8.602
road	-0.99	5.315

These two-dimensional word vectors are invented for teaching. Real embeddings have hundreds or thousands of dimensions.

Start at the default query [2, 2]: a long vector (dog) points exactly its way, a short one (pet) sits right next to it. Then try the normalize box.

Predict first. You normalize the query and every word to length 1, then rank by Euclidean distance. How does that ranking compare with ranking by cosine?

Closeness is not truth

Similarity depends on the representation (which model produced the vectors), on normalization (what you do about length) and on the task. Even when everything is done right, a small distance only says the two items look alike to that model. A claim and its negation are often about the same topic, so a model can place them close together. Embedding search finds related text; it does not check whether the text is correct.

Chapter context

A vector is an ordered set of coordinates in a specified space. It can describe a displacement, a data record or a learned representation. Operations are meaningful only when the coordinates refer to compatible dimensions.

TRY IT YOURSELF

Part 1. Do it on the plane

The query starts at [-2, -2], where cosine and distance happen to agree. Your job is to find where they do not.

normalize to length 1

Both measures pick road.

word	cosine	distance
road ◯c ◯d	0.99	0.5
car	0.97	3.162
bone	0.196	3.041
love	-0.141	6.083
hug	-0.196	3.64
fish	-0.394	7
pet	-0.894	5.831
dog	-0.999	10.966

These two-dimensional word vectors are invented for teaching. Real embeddings have hundreds or thousands of dimensions.

To do: Drag the query so that cosine similarity and Euclidean distance pick different nearest words
To do: At a query where they disagree, switch on length normalization and watch them agree
To do: Make cosine pick love while distance picks hug (normalization off)

Part 2. Do it in code

The dictionary VOCAB holds the same toy 2D word vectors as the widget. Write normalize(v), nearest_cosine(q, vocab) and nearest_distance(q, vocab); each nearest function returns the word.
Then use them: after normalizing the query and every word, nearest_distance must agree with nearest_cosine.

import numpy as np

VOCAB = {
    "dog": [6, 5.5],
    "pet": [1, 3],
    "fish": [5, -2],
    "bone": [1, -1.5],
    "car": [-5, -3],
    "road": [-2, -1.5],
    "love": [-3, 4],
    "hug": [-1, 1.5],
}

def normalize(v):
    # scale v to length 1; ValueError for the zero vector
    pass

def nearest_cosine(q, vocab):
    # the word whose vector has the highest cosine similarity with q
    pass

def nearest_distance(q, vocab):
    # the word whose vector has the smallest Euclidean distance to q
    pass

print(nearest_cosine([2, 2], VOCAB), nearest_distance([2, 2], VOCAB))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can two embeddings be close while describing false claims?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

a\cdot b=\sum_i a_i b_i,\quad \|a\|_2=\sqrt{\sum_i a_i^2}.

\cos(a,b)=\frac{a\cdot b}{\|a\|_2\|b\|_2},\quad \operatorname{proj}_b a=\frac{a\cdot b}{b\cdot b}b.

Both vectors must have the same dimension; cosine and projection need the denominator to be nonzero.

Worked example

For a=[2,3] and b=[4,1], matching products are 8 and 3, so the dot product is 11. Squared norms are 13 and 17. The cosine is 11/sqrt(221), approximately 0.73994. The projection of [3,4] onto [1,0] keeps only the first coordinate.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
from operator import mul
a, b = [2, 3], [4, 1]
dot = sum(map(mul, a, b))
assert dot == 11
cosine = dot / (sqrt(sum(x*x for x in a))*sqrt(sum(x*x for x in b)))
assert isclose(cosine, 11/sqrt(221))
assert isclose(cosine, 0.7399400734, abs_tol=1e-9)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Dot product is not elementwise multiplication alone. Similarity and distance have opposite ordering conventions. Zero vectors cannot be normalized by dividing by their norm.