D8.6 · LESSON 6 OF 6

Adaptation and evaluation

Compare learned embeddings, transfer learning, fine-tuning and contrastive training objectives.

THE IDEA

Adapt cheaply, compare fairly

There is a ladder of adaptation: use frozen features with a small head; train small add-on pieces such as low-rank adapters (a rank-r update B·A added to a weight matrix, costing r·(d_in + d_out) numbers instead of d_in·d_out); or tune everything. Each costs more and risks more forgetting.

To compare them honestly, change one thing at a time. A bigger head can raise accuracy even if the encoder is no better, so judging an encoder requires the same head on every side.

setup	encoder	head	accuracy
random frozen encoder + MLP head	random	mlp	0.78
pretrained frozen encoder + linear head	pretrained	linear	0.74
pretrained frozen encoder + MLP head	pretrained	mlp	0.83

Pretrained+MLP (0.83) beats pretrained+linear (0.74), but those rows differ in the head and nothing else, so they say nothing about the encoder. Only rows with the same head isolate it.

adapter size for one 100×100 matrix at rank 8: 8·(100+100) = 1600 10 matrices: 16000 vs 100000 to train them fully

Three results. Which pair of rows is a fair test of the encoder?

Predict first. Does a larger head prove the encoder improved?

NumPy: an adapted layer

y = W @ x + scale * (B @ (A @ x))   # W frozen; only A (r x d_in) and B (d_out x r) train

Chapter context

A representation is useful when its structure supports a downstream task. Learning a representation changes which examples appear similar, but those similarities inherit assumptions from training data and objectives.

TRY IT YOURSELF

Part 1. Do it hands-on

Find the fair comparison in a results table and count adapter parameters. In Python, write the adapter maths and a comparison that refuses unfair pairs.

setup	encoder	head	accuracy
random frozen encoder + MLP head	random	mlp	0.78
pretrained frozen encoder + linear head	pretrained	linear	0.74
pretrained frozen encoder + MLP head	pretrained	mlp	0.83

Pretrained+MLP (0.83) beats pretrained+linear (0.74), but those rows differ in the head and nothing else, so they say nothing about the encoder. Only rows with the same head isolate it.

adapter size for one 100×100 matrix at rank 8: 8·(100+100) = 1600 10 matrices: 16000 vs 100000 to train them fully

gain from pretraining, same head adapter parameters for 10 matrices percent of full tuning

To do: Results: random frozen encoder + MLP head 0.78; pretrained frozen + linear head 0.74; pretrained frozen + MLP head 0.83. Which two rows isolate the encoder (same head)? Type the accuracy gain from pretraining (2 decimals)
To do: Low-rank adapters of rank 8 on a 100×100 matrix add r·(100 + 100) numbers. Type the total for 10 such matrices
To do: Full tuning would train all 100×100×10 = 100,000 weights. What fraction of that do the adapters train? Type it as a percentage number

Part 2. Do it in code

lora_params(d_in, d_out, r): parameters of a rank-r adapter: r * (d_in + d_out). r must be at least 1 (ValueError).
trainable_fraction(trainable, total): trainable / total as a percentage (0 to 100). total must be positive (ValueError).
lora_forward(W, A, B, x, scale=1.0): the adapted layer output W @ x + scale * (B @ (A @ x)). W is (d_out, d_in), A is (r, d_in), B is (d_out, r).
fair_gain(rows, a, b): rows is a list of dicts with keys name, acc, head. Return rows[a].acc − rows[b].acc only if both rows use the same head; otherwise raise ValueError (the comparison would mix encoder and head).

import numpy as np

def lora_params(d_in, d_out, r):
    pass

def trainable_fraction(trainable, total):
    pass

def lora_forward(W, A, B, x, scale=1.0):
    pass

def fair_gain(rows, a, b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a larger head prove the encoder improved?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One contrastive loss for an anchor a and positive p is

L=-\log\frac{\exp(s(a,p)/\tau)}{\sum_{j\in\mathcal C}\exp(s(a,j)/\tau)}.

The candidate set includes the designated positive; other candidate definitions depend on the method. Temperature tau is positive and controls concentration.

Worked example

For a candidate set with positive logit ln 2 and one alternative logit 0, exponentiated weights are 2 and 1. The positive probability is 2/3 and loss is −ln(2/3)≈0.405465. This demonstrates the normalization; actual contrastive batches usually contain many more candidates.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
logits=[log(2),0]
prob=exp(logits[0])/sum(exp(v) for v in logits)
assert isclose(prob,2/3)
assert isclose(-log(prob),.40546510810816444)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Embedding proximity is task-dependent. Frozen parameters can coexist with a trainable head. Parameter-efficient tuning still requires evaluation for forgetting and task failures.