L7.3 · LESSON 3 OF 6

RLHF and constraints

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Optimising a reward without drifting too far

RLHF uses reinforcement learning to make the model produce answers the reward model scores highly. Left alone, the optimiser will exploit the reward model and change behaviour far from the starting model. So the objective subtracts a penalty for drift from a reference policy (usually the SFT model): reward − β · log(π / π_ref).

penalty weight β0.5

answer	reward r	log-ratio log π − log π_ref	r − β·log-ratio
stays close to the start	3	0.5	2.75
drifted, high reward	5	6	2

the optimiser prefers: stays close to the start With β = 0 only the reward counts. A larger β makes straying from the reference policy expensive.

A real RLHF run has more parts: collecting preference data, training the reward model, the RL algorithm itself, and evaluation on behaviour the reward does not cover.

Move β. A higher penalty weight makes the drifted high-reward answer less attractive.

Predict first. Why monitor drift from the starting behaviour?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute penalised rewards and choose a β that discourages drift, then write the penalty and selection code.

penalty weight β0

answer	reward r	log-ratio log π − log π_ref	r − β·log-ratio
stays close to the start	3	0.5	3
drifted, high reward	5	6	5

the optimiser prefers: drifted, high reward With β = 0 only the reward counts. A larger β makes straying from the reference policy expensive.

A real RLHF run has more parts: collecting preference data, training the reward model, the RL algorithm itself, and evaluation on behaviour the reward does not cover.

reward 2, log-ratio 1.5, β 0.5 reward 5, log-ratio 6, β 0.5 The slider above is for the third task.

To do: Penalised reward = reward − β · log-ratio, where the log-ratio is log π(answer) − log π_ref(answer). Reward 2, log-ratio 1.5, β = 0.5. Type it
To do: A drifted answer: reward 5, log-ratio 6, β = 0.5. Type its penalised reward
To do: Choose β so the drifted answer (reward 5, log-ratio 6) scores LOWER than the safe answer (reward 3, log-ratio 0.5)

Part 2. Do it in code

log_ratio(logp, logp_ref): logp − logp_ref (works for numbers and arrays).
penalized_reward(r, logp, logp_ref, beta): r − beta · log_ratio. beta must be >= 0 (ValueError).
best_sample(samples, beta): samples is a list of dicts with r, logp, logp_ref. Return the one with the highest penalised reward.
mean_drift(logps, logp_refs): the average log-ratio over samples drawn from the policy (an estimate of KL(policy ‖ reference)). ValueError for an empty list.
min_beta(a, b): the smallest beta above which sample a (dict with r, lr) scores lower than sample b (dict with r, lr) when a has the higher reward but more drift. Return None if a never scores lower (when a['lr'] <= b['lr']).

import numpy as np

def log_ratio(logp, logp_ref):
    pass

def penalized_reward(r, logp, logp_ref, beta):
    pass

def best_sample(samples, beta):
    pass

def mean_drift(logps, logp_refs):
    pass

def min_beta(a, b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why monitor drift from the starting behavior?

ADrift is always a sign of success.BOptimizing one reward can degrade behaviors the reward does not measure, and the drift penalty and separate evaluations keep that visible.CDrift makes training faster, so it should be maximised.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.