L7.1 · LESSON 1 OF 6

Supervised fine-tuning

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Teaching by example

A pretrained model continues text, but it does not necessarily answer questions. Supervised fine-tuning (SFT) continues training on selected (prompt, response) pairs so it learns to respond in the wanted way. The training loss is the same next-token loss, with one key choice: usually only the response tokens count. The prompt is context, not something to imitate.

Q:2+2?A:4<eos>

Grey: prompt (the user's side). Green: the response we want the model to imitate. Each position below predicts the token to its right; click to include its loss.

position predicts	loss	in the loss?
2 (prompt)	2
+ (prompt)	2
2 (prompt)	2
? (prompt)	2
A: (prompt)	2
4 (response)	0.8
<eos> (response)	0.2

average over the included positions = 0.5 SFT usually trains only on the response, so the model learns to answer, not to imitate the question.

Click positions to include or exclude them from the loss.

Predict first. Does a polished answer automatically make a good reference example for every task?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Count graded positions and compare the masked and unmasked loss, then build SFT examples with prompt masking.

Q:2+2?A:4<eos>

Grey: prompt (the user's side). Green: the response we want the model to imitate. Each position below predicts the token to its right; click to include its loss.

position predicts	loss	in the loss?
2 (prompt)	2
+ (prompt)	2
2 (prompt)	2
? (prompt)	2
A: (prompt)	2
4 (response)	0.8
<eos> (response)	0.2

average over the included positions = 0.5 SFT usually trains only on the response, so the model learns to answer, not to imitate the question.

graded predictions (response only) mean over the response mean over all seven (3 decimals)

To do: Tokens: Q: 2 + 2 ? A: 4 <eos>. The first 6 are the prompt, the last 2 the response. With the loss only on the response, how many next-token predictions are graded?
To do: The seven next-token losses are [2, 2, 2, 2, 2, 0.8, 0.2]. Average over the response positions only
To do: Average over ALL seven positions (3 decimals)

Part 2. Do it in code

build_example(prompt, response, eos="<eos>"): tokens = prompt + response + [eos]. labels has the same length: -100 for every prompt token, and the token itself for every response token and for eos (those are the graded positions).
graded_count(labels): how many labels are not -100.
masked_loss(losses, labels): average of the per-position losses at graded positions only. losses has the same length as labels. ValueError if nothing is graded.
keep_example(example_tokens, min_len, max_len, banned): True if the number of tokens is between min_len and max_len (inclusive) and no banned word occurs in the tokens.

def build_example(prompt, response, eos="<eos>"):
    pass

def graded_count(labels):
    pass

def masked_loss(losses, labels):
    pass

def keep_example(example_tokens, min_len, max_len, banned):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a polished answer automatically make a good reference example for every task?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.