L7.6 · LESSON 6 OF 6

Distillation and reasoning training

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Learning from a teacher

Distillation trains a (usually smaller) student to match a teacher's output distribution instead of, or in addition to, hard labels. The soft probabilities carry more information than a single correct answer: they show how the teacher ranks the wrong ones. A temperature T above 1 flattens the teacher so those differences are easier to learn.

temperature T1teacher logit 12teacher logit 20.5teacher logit 3-1

class	teacher softmax(logits / T)	student (fixed)
1	0.786	0.5
2	0.175	0.3
3	0.039	0.2

soft cross-entropy = 0.8185 teacher entropy = 0.6216 KL = 0.1969 Higher T flattens the teacher so the student also learns how it ranks the wrong answers.

The student is trained to match the teacher, so it can inherit the teacher's mistakes along with its skills. Reasoning-focused training adds task feedback and evaluation; longer outputs alone do not show better reasoning.

Change the teacher's logits and the temperature, and watch the divergence from a fixed student.

Predict first. Can a student inherit a teacher's mistakes?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute distillation quantities by hand, then write the temperature softmax, KL and a combined distillation loss.

temperature T1teacher logit 12teacher logit 20.5teacher logit 3-1

class	teacher softmax(logits / T)	student (fixed)
1	0.786	0.5
2	0.175	0.3
3	0.039	0.2

soft cross-entropy = 0.8185 teacher entropy = 0.6216 KL = 0.1969 Higher T flattens the teacher so the student also learns how it ranks the wrong answers.

CE for [0.7,0.2,0.1] vs [0.5,0.3,0.2] KL first class, logits [2,0], T = 2

To do: Teacher probabilities [0.7, 0.2, 0.1], student [0.5, 0.3, 0.2]. Soft cross-entropy −Σ teacher·ln(student) (3 decimals)
To do: KL(teacher ‖ student) = cross-entropy − teacher entropy (entropy 0.802). Type it (3 decimals)
To do: Temperature softens a teacher. Logits [2, 0] at T = 2 are divided to [1, 0]. Type the probability of the first class (3 decimals)

Part 2. Do it in code

softmax_t(logits, T): softmax of logits / T (stable). T must be positive (ValueError).
soft_cross_entropy(t, s): −Σ t·ln s for probability vectors (skip terms with t = 0).
kl_div(t, s): KL(t ‖ s) = Σ t·ln(t/s) (terms with t = 0 contribute 0).
distill_loss(student_logits, teacher_logits, hard_label, T=2.0, alpha=0.5): alpha · T² · KL(softmax_t(teacher, T) ‖ softmax_t(student, T)) + (1 − alpha) · (−ln softmax(student)[hard_label]).

import numpy as np

def softmax_t(logits, T):
    pass

def soft_cross_entropy(t, s):
    pass

def kl_div(t, s):
    pass

def distill_loss(student_logits, teacher_logits, hard_label, T=2.0, alpha=0.5):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a student inherit a teacher's mistakes?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.