D8.5 · LESSON 5 OF 6

Contrastive learning

Compare learned embeddings, transfer learning, fine-tuning and contrastive training objectives.

THE IDEA

Pull matching pairs together, push the rest away

Contrastive learning trains an encoder so that two views of the same item (a positive pair) get a high similarity, while views of different items (the negatives) get a lower one. For one anchor, softmax turns all the candidate similarities into probabilities, and the loss is −ln of the positive's probability.

The worked example: positive logit ln 2 and one alternative with logit 0 give weights 2 and 1, so the positive has probability 2/3 and the loss is −ln(2/3) ≈ 0.405. Real batches have many more alternatives.

positive logit0.69number of alternatives3

candidate	logit	weight e^logit	probability
positive (the other view of the same item)	0.69	2	0.4
alternative 1	0	1	0.2
alternative 2	0	1	0.2
alternative 3	0	1	0.2

loss = −ln P(positive) = −ln 0.4 = 0.9163 lower loss = the positive is pulled toward its anchor RELATIVE to the alternatives

Change the positive logit and the number of alternatives.

Predict first. Could an augmentation accidentally erase the feature needed for the downstream task?

Chapter context

A representation is useful when its structure supports a downstream task. Learning a representation changes which examples appear similar, but those similarities inherit assumptions from training data and objectives.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute the loss for different numbers of alternatives, then raise the positive until the loss is small. In Python, write InfoNCE and a batch version.

positive logit0.69number of alternatives3

candidate	logit	weight e^logit	probability
positive (the other view of the same item)	0.69	2	0.4
alternative 1	0	1	0.2
alternative 2	0	1	0.2
alternative 3	0	1	0.2

loss = −ln P(positive) = −ln 0.4 = 0.9163 lower loss = the positive is pulled toward its anchor RELATIVE to the alternatives

P(positive), 1 alternative loss loss with 3 alternatives

positive logit (3 alternatives, goal loss < 0.1)0

loss at logit 0 with 3 alternatives: 1.3863

To do: One positive with logit ln 2 and one alternative with logit 0: weights are 2 and 1. Type the positive's probability (3 decimals)
To do: The loss is −ln(probability). Type it (3 decimals)
To do: Now three alternatives, all with logit 0, and the positive still at logit ln 2 (weights 2, 1, 1, 1). Type the loss (3 decimals)
To do: With those three alternatives (logit 0), raise the positive's logit until the loss is below 0.1

Part 2. Do it in code

info_nce(pos, negs, tau=1.0): one anchor. pos is the similarity with its positive, negs the similarities with the alternatives. Logits are similarity / tau. Return −log of the positive's softmax probability (stable: subtract the maximum). tau must be positive (ValueError).
info_nce_batch(S, tau=1.0): S is an (n, n) similarity matrix where entry [i, i] is the positive pair of anchor i and the other entries in the row are negatives. Return the mean info_nce over the rows.
similarity_matrix(A, B): cosine similarity between every row of A and every row of B, shape (len(A), len(B)).

import numpy as np

def info_nce(pos, negs, tau=1.0):
    pass

def info_nce_batch(S, tau=1.0):
    pass

def similarity_matrix(A, B):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Could an augmentation accidentally erase the feature needed for the downstream task?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One contrastive loss for an anchor a and positive p is

L=-\log\frac{\exp(s(a,p)/\tau)}{\sum_{j\in\mathcal C}\exp(s(a,j)/\tau)}.

The candidate set includes the designated positive; other candidate definitions depend on the method. Temperature tau is positive and controls concentration.

Worked example

For a candidate set with positive logit ln 2 and one alternative logit 0, exponentiated weights are 2 and 1. The positive probability is 2/3 and loss is −ln(2/3)≈0.405465. This demonstrates the normalization; actual contrastive batches usually contain many more candidates.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
logits=[log(2),0]
prob=exp(logits[0])/sum(exp(v) for v in logits)
assert isclose(prob,2/3)
assert isclose(-log(prob),.40546510810816444)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding proximity is task-dependent. Frozen parameters can coexist with a trainable head. Parameter-efficient tuning still requires evaluation for forgetting and task failures.