L7.2 · LESSON 2 OF 6

Human preferences and rewards

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Learning what people prefer

Some qualities (helpful, polite, safe) are easier to compare than to write down. For a prompt, two answers are shown to people who say which is better under a rubric. A reward model is trained to score answers so that the preferred one scores higher: it predicts the preference with sigmoid(score difference), and the loss is −ln of that probability.

reward of the preferred answer2reward of the rejected answer1

P(preferred wins) = sigmoid(2 − 1) = 0.7311 pair loss = −ln P = 0.3133 (0.693 when the scores are equal; smaller as the model ranks the pair correctly)

A reward model learns from many such comparisons. It is a learned stand-in for human judgement, not truth:

answer	learned reward	actually correct?
A: long, confident, wrong	3	no
B: short, correct	2	yes

optimising the reward picks: A: long, confident, wrong → wrong: the optimiser found something the reward model over-values

Set two scores and read the loss. Then look at what optimising a learned score can do.

Predict first. Does a high learned reward prove an answer is true?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute a preference probability and loss, then write the reward-model loss and see a high-reward wrong answer.

reward of the preferred answer2reward of the rejected answer1

P(preferred wins) = sigmoid(2 − 1) = 0.7311 pair loss = −ln P = 0.3133 (0.693 when the scores are equal; smaller as the model ranks the pair correctly)

A reward model learns from many such comparisons. It is a learned stand-in for human judgement, not truth:

answer	learned reward	actually correct?
A: long, confident, wrong	3	no
B: short, correct	2	yes

optimising the reward picks: A: long, confident, wrong → wrong: the optimiser found something the reward model over-values

P(preferred wins) for 2.0 vs 1.0 pair loss loss when the scores are equal

To do: A reward model scores the preferred answer 2.0 and the rejected one 1.0. The probability it assigns to "preferred wins" is sigmoid(2.0 − 1.0). Type it (3 decimals)
To do: The training loss for that pair is −ln of that probability (3 decimals)
To do: If the model scores both answers equally, the loss is…

Part 2. Do it in code

sigmoid(z) = 1 / (1 + exp(−z)).
preference_prob(r_chosen, r_rejected): sigmoid(r_chosen − r_rejected), the model's probability that the chosen answer wins.
pairwise_loss(r_chosen, r_rejected): the mean of −ln preference_prob over arrays of rewards (use a stable form: log1p(exp(−diff))).
pair_accuracy(r_chosen, r_rejected): the fraction of pairs where the chosen answer has the higher reward.
best_by_reward(cands): cands is a list of dicts with reward and correct. Return the one with the highest reward (first on ties).

import numpy as np

def sigmoid(z):
    pass

def preference_prob(r_chosen, r_rejected):
    pass

def pairwise_loss(r_chosen, r_rejected):
    pass

def pair_accuracy(r_chosen, r_rejected):
    pass

def best_by_reward(cands):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a high learned reward prove an answer is true?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.