L7.5 · LESSON 5 OF 6

LoRA and QLoRA

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

THE IDEA

Adapting with tiny add-ons

Updating every weight of a large model is expensive. LoRA freezes the base matrix W and learns a low-rank update B·A, with A of size r × d_in and B of size d_out × r. It adds r·(d_in + d_out) trainable numbers instead of d_in·d_out. For a 512×512 matrix at rank 8 that is 8,192 instead of 262,144, a 32-fold reduction in what is trained. The update does not need to touch every entry independently; it is constrained to rank at most r.

QLoRA additionally stores the frozen base weights in a low-precision (quantised) format to save memory, while the adapters are trained in higher precision. The base still occupies space; it is just smaller.

matrix size d (d × d)512rank r8

frozen base W: 512 × 512 = 262,144 weights (not updated) adapter: A is 8 × 512, B is 512 × 8 → 8,192 trainable weights update = B·A has the full 512 × 512 shape but only rank ≤ 8 trainable share: 3.125% (32× fewer)

base weights stored as	bytes for this matrix
16 bits	524,288
8 bits	262,144
4 bits (QLoRA-style quantised base)	131,072

The adapters are tiny, but the frozen base still occupies memory; quantising it is what the QLoRA recipe adds. At the start B is zero, so the adapted model equals the base model.

Change the matrix size and the rank.

Predict first. Does LoRA require updating every entry of the base matrix?

Chapter context

A pretrained model predicts text; post-training adapts its behavior for intended tasks. Data quality, preference definitions and evaluation matter as much as the optimization label. Different methods change different parameters or training signals.

TRY IT YOURSELF

Part 1. Do it hands-on

Count adapter weights and quantised storage, then write a LoRA layer and prove that merging it gives the same outputs.

matrix size d (d × d)512rank r8

base weights stored as	bytes for this matrix
16 bits	524,288
8 bits	262,144
4 bits (QLoRA-style quantised base)	131,072

The adapters are tiny, but the frozen base still occupies memory; quantising it is what the QLoRA recipe adds. At the start B is zero, so the adapted model equals the base model.

rank-8 adapter on 512 × 512 times fewer than full bytes for the base at 4 bits

To do: A 512×512 matrix has 262,144 weights. A rank-8 LoRA adds A (8×512) and B (512×8). How many trainable adapter weights?
To do: How many times fewer trainable weights than updating the full matrix?
To do: QLoRA stores the frozen base weights in 4 bits. Bytes for the 262,144 base weights at 4 bits each (ignoring scale factors)?

Part 2. Do it in code

lora_params(d_in, d_out, r): r·(d_in + d_out). r must be at least 1 (ValueError).
lora_forward(W, A, B, x, alpha=None): W @ x + (alpha / r) · B @ (A @ x), with r = A.shape[0]. If alpha is None use alpha = r (scale 1).
merge(W, A, B, alpha=None): the merged weight W + (alpha / r) · B @ A. It must give the same outputs as lora_forward.
quantized_bytes(n_params, bits): bytes needed to store n_params values at `bits` bits each (n_params · bits / 8, rounded up to a whole byte).
trainable_fraction(d_in, d_out, r): the adapter size divided by d_in·d_out.

import numpy as np

def lora_params(d_in, d_out, r):
    pass

def lora_forward(W, A, B, x, alpha=None):
    pass

def merge(W, A, B, alpha=None):
    pass

def quantized_bytes(n_params, bits):
    pass

def trainable_fraction(d_in, d_out, r):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does LoRA require updating every entry of the base matrix?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

LoRA writes $W'=W+BA$ , with $A\in\mathbb R^{r\times d_{in}}$ and $B\in\mathbb R^{d_{out}\times r}$ . Adapter weights total $r(d_{in}+d_{out})$ . A basic DPO loss is $-\log\sigma(\beta\Delta)$ , where Delta is the chosen-minus-rejected difference of policy/reference log-ratios.

Writing the DPO log-ratio difference explicitly:

\Delta=\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}.

Here x is the prompt, y-w the preferred response, y-l the rejected response and beta the loss scale. Sequence log probabilities are sums under a consistent response-token convention.

Worked example

A 512×512 matrix has 262,144 weights. A rank-eight LoRA update uses 8×512 + 512×8 = 8,192 trainable adapter weights, a 32-fold reduction for this matrix's trainable update. The frozen base still occupies storage. With a DPO log-ratio difference of zero, the loss is ln 2≈0.693147.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,isclose
full=512*512
adapter=8*(512+512)
assert full == 262144 and adapter == 8192
assert full/adapter == 32
assert isclose(-log(.5),.6931471805599453)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fewer trainable weights do not mean the frozen model disappears. Preference agreement and factual correctness are different measures. QLoRA is more specific than any use of low-bit weights.