L7.4 · LESSON 4 OF 6

Direct preference optimization

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Preferences without a separate reward model

Direct preference optimization (DPO) trains on (chosen, rejected) pairs directly. For each answer it looks at how much more likely the policy has made it compared with the frozen reference, then asks for the chosen answer to have moved up more than the rejected one:

loss = −ln sigmoid( β · [ (log π(c) − log π_ref(c)) − (log π(r) − log π_ref(r)) ] )

If the policy equals the reference the bracket is 0 and the loss is ln 2 ≈ 0.693.

chosen log-ratio (policy vs reference)2rejected log-ratio-1β0.5

margin = β·(chosen − rejected) = 0.5·(2 − -1) = 1.5 loss = −ln sigmoid(margin) = 0.2014

DPO trains a classification-style loss directly on (chosen, rejected) pairs, measured against a frozen reference policy: it pushes the chosen answer up and the rejected one down relative to the reference. No separate reward model is trained in the basic form, but the result is only as good as the preference data. It is not the same as SFT on the chosen answer alone, because the rejected answer and the reference both enter the loss.

Move the two log-ratios. The loss falls as the chosen answer gains more than the rejected one.

Predict first. Is DPO identical to ordinary SFT on only the chosen response?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute a DPO margin and loss by hand, then write the loss and implicit reward in Python.

chosen log-ratio (policy vs reference)2rejected log-ratio-1β0.5

margin = β·(chosen − rejected) = 0.5·(2 − -1) = 1.5 loss = −ln sigmoid(margin) = 0.2014

loss when the margin is 0 margin for +2, −1, β 0.5 loss at that margin

To do: DPO loss = −ln sigmoid(β·[(chosen log-ratio) − (rejected log-ratio)]). If the difference of log-ratios is 0 the loss is…
To do: Chosen log-ratio = +2 (the policy now likes it more than the reference), rejected log-ratio = −1, β = 0.5. Type the margin β·(2 − (−1))
To do: The loss for that margin: −ln sigmoid(1.5) (3 decimals)

Part 2. Do it in code

dpo_margin(lp_c, lp_r, ref_c, ref_r, beta): beta · ((lp_c − ref_c) − (lp_r − ref_r)) for log-probabilities of the chosen and rejected answers under the policy and the reference (works on arrays).
dpo_loss(lp_c, lp_r, ref_c, ref_r, beta): mean of −ln sigmoid(margin), computed stably.
implicit_reward(lp, ref, beta): beta · (lp − ref).
preference_accuracy(lp_c, lp_r, ref_c, ref_r): fraction of pairs where the implicit reward of the chosen answer is higher.

import numpy as np

def dpo_margin(lp_c, lp_r, ref_c, ref_r, beta):
    pass

def dpo_loss(lp_c, lp_r, ref_c, ref_r, beta):
    pass

def implicit_reward(lp, ref, beta):
    pass

def preference_accuracy(lp_c, lp_r, ref_c, ref_r):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is DPO identical to ordinary SFT on only the chosen response?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.