D8.4 · LESSON 4 OF 6

Self-supervision

Compare learned embeddings, transfer learning, fine-tuning and contrastive training objectives.

THE IDEA

Labels from the data itself

Labelled data is expensive; raw text, images and audio are plentiful. Self-supervised learning invents the training task from the data: hide something and predict it. There is still an objective and a target; the target just comes from the data itself instead of a person.

input : the cat [MASK] on the mat target: sat (comes from the data itself, nobody labelled it)

The way the target is built decides what the model must learn: masking forces use of both sides of the context, next-word prediction forces it to anticipate.

Two ways to build targets from one sentence.

The choice matters. Predicting a masked word uses context on both sides; predicting the next word only the left. For images, building two augmented views of one picture teaches invariance to the augmentations (crops, colour shifts).

Predict first. Does self-supervision mean training without any objective or target?

Chapter context

A representation is useful when its structure supports a downstream task. Learning a representation changes which examples appear similar, but those similarities inherit assumptions from training data and objectives.

TRY IT YOURSELF

Part 1. Do it hands-on

Build masked and next-word targets by hand, then write the target-construction functions in Python.

input : the cat [MASK] on the mat target: sat (comes from the data itself, nobody labelled it)

The way the target is built decides what the model must learn: masking forces use of both sides of the context, next-word prediction forces it to anticipate.

mask the 3rd word:

next-word pairs from 6 words words to predict when masking 15% of 20

To do: Hide the third word of "the cat sat on the mat" with [MASK]. Type the six input words
To do: Next-word prediction on a 6-word sentence: each word (except the last) is an input and the following word is its target. How many (input, target) pairs?
To do: If you mask 15% of a 20-word passage, about how many words does the model have to predict?

Part 2. Do it in code

next_token_pairs(tokens): list of (input, target) pairs where the target is the token after the input.
mask_tokens(tokens, positions, mask="[MASK]"): return (inputs, targets). inputs is a new list with the given positions replaced by mask; targets is a dict position -> original token. Raise ValueError for a position out of range.
random_mask(tokens, frac, seed): choose k = max(1, round(frac * n)) distinct positions with np.random.default_rng(seed).choice(n, k, replace=False), sort them and apply mask_tokens. Return (inputs, targets).
crop_view(x, start, size): a view of a 1-D list from start of length size. Raise ValueError if it does not fit.

import numpy as np

def next_token_pairs(tokens):
    pass

def mask_tokens(tokens, positions, mask="[MASK]"):
    pass

def random_mask(tokens, frac, seed):
    pass

def crop_view(x, start, size):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does self-supervision mean training without any objective or target?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One contrastive loss for an anchor a and positive p is

L=-\log\frac{\exp(s(a,p)/\tau)}{\sum_{j\in\mathcal C}\exp(s(a,j)/\tau)}.

The candidate set includes the designated positive; other candidate definitions depend on the method. Temperature tau is positive and controls concentration.

Worked example

For a candidate set with positive logit ln 2 and one alternative logit 0, exponentiated weights are 2 and 1. The positive probability is 2/3 and loss is −ln(2/3)≈0.405465. This demonstrates the normalization; actual contrastive batches usually contain many more candidates.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
logits=[log(2),0]
prob=exp(logits[0])/sum(exp(v) for v in logits)
assert isclose(prob,2/3)
assert isclose(-log(prob),.40546510810816444)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding proximity is task-dependent. Frozen parameters can coexist with a trainable head. Parameter-efficient tuning still requires evaluation for forgetting and task failures.