D8.1 · LESSON 1 OF 6

Embeddings

Compare learned embeddings, transfer learning, fine-tuning and contrastive training objectives.

THE IDEA

Items as points in a space

An embedding turns an item (a word, an image, a user) into a vector of numbers. The numbers are learned so that items that behave alike end up close together. Similarity is usually measured by the cosine of the angle between two vectors: 1 for the same direction, 0 for unrelated, −1 for opposite. It ignores length.

nearest to

word	vector	cosine with cat
cat	[1, 0.2]	1
dog	[0.9, 0.3]	0.992
car	[0, 1]	0.196
truck	[0.1, 0.9]	0.303

Rotating all vectors changes every coordinate but no cosine. What an embedding means is in the relationships, not in "the first coordinate".

Four words in two dimensions. Pick a word to rank the others. Then rotate the whole space.

Predict first. Must the first coordinate always mean the same human concept across two separately trained models?

NumPy

E = table[ids]                                   # an embedding layer is just row lookup
cos = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

Chapter context

A representation is useful when its structure supports a downstream task. Learning a representation changes which examples appear similar, but those similarities inherit assumptions from training data and objectives.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute similarities by hand and see that rotation changes nothing. Then write cosine, lookup, nearest-neighbour search and an analogy query.

nearest to

word	vector	cosine with cat
cat	[1, 0.2]	1
dog	[0.9, 0.3]	0.992
car	[0, 1]	0.196
truck	[0.1, 0.9]	0.303

Rotating all vectors changes every coordinate but no cosine. What an embedding means is in the relationships, not in "the first coordinate".

cosine(cat, dog) nearest neighbour of truck cosine after the 90° rotation

To do: Embeddings cat = [1, 0.2] and dog = [0.9, 0.3]. Type their cosine similarity (3 decimals)
To do: Which word is the nearest neighbour of "truck" = [0.1, 0.9] by cosine? Type it
To do: Rotate every vector by 90° (the first coordinate now means something different). Type the cosine of the rotated cat and dog

Part 2. Do it in code

cosine(a, b): dot(a, b) / (|a| |b|). Raise ValueError if either vector has length 0.
lookup(table, ids): table has shape (vocab, d). Return the rows for the integer ids as an array (len(ids), d). Raise IndexError-like errors for out-of-range ids (a ValueError is fine).
nearest(query, E, names, k=1, exclude=None): the k names whose rows of E have the highest cosine similarity with query, best first. If exclude (a name) is given, leave it out.
analogy(a, b, c, E, names): the name nearest to E[b] − E[a] + E[c], excluding a, b and c.

import numpy as np

def cosine(a, b):
    pass

def lookup(table, ids):
    pass

def nearest(query, E, names, k=1, exclude=None):
    pass

def analogy(a, b, c, E, names):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must the first coordinate always mean the same human concept across trained models?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One contrastive loss for an anchor a and positive p is

L=-\log\frac{\exp(s(a,p)/\tau)}{\sum_{j\in\mathcal C}\exp(s(a,j)/\tau)}.

The candidate set includes the designated positive; other candidate definitions depend on the method. Temperature tau is positive and controls concentration.

Worked example

For a candidate set with positive logit ln 2 and one alternative logit 0, exponentiated weights are 2 and 1. The positive probability is 2/3 and loss is −ln(2/3)≈0.405465. This demonstrates the normalization; actual contrastive batches usually contain many more candidates.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
logits=[log(2),0]
prob=exp(logits[0])/sum(exp(v) for v in logits)
assert isclose(prob,2/3)
assert isclose(-log(prob),.40546510810816444)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding proximity is task-dependent. Frozen parameters can coexist with a trainable head. Parameter-efficient tuning still requires evaluation for forgetting and task failures.