L6.2 · LESSON 2 OF 6

Targets and loss

Describe a pretraining loop and reason about data, batch size, precision and a simplified compute budget.

THE IDEA

What exactly is being graded

At every position the model predicts the next token, so a sequence of n tokens gives n − 1 graded positions. The loss at a position is −ln of the probability the model gave to the true next token, and the training loss is the average over graded positions.

Batches are rectangular, so shorter sequences are padded. Padding is not text, so those positions must be excluded from the average. Otherwise the loss includes meaningless terms and the numbers (and gradients) are wrong.

position	input	target	loss
1	the	cat	1
2	cat	sat	2
3	sat	down	3
4	down	(end)	4
5	<pad>	<pad>	9
6	<pad>	<pad>	9

loss over counted positions = 2.5 loss over all 6 positions = 4.667 (padding wrongly treated as supervision)

A causal mask keeps each position from seeing later tokens; the loss mask keeps padding from being graded. They are different masks for different jobs.

Click "yes/no" to include or exclude each position.

Predict first. Can the loss include padded positions without a defined target policy?

Chapter context

Pretraining optimizes next-token predictions over a dataset. Its costs depend on the architecture, sequence lengths, implementation and hardware. A useful budget model states which work it approximates and which work it ignores.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute the loss with and without padding, then write target shifting, the loss mask and a masked cross-entropy.

position	input	target	loss
1	the	cat	1
2	cat	sat	2
3	sat	down	3
4	down	(end)	4
5	<pad>	<pad>	9
6	<pad>	<pad>	9

loss over counted positions = 2.5 loss over all 6 positions = 4.667 (padding wrongly treated as supervision)

A causal mask keeps each position from seeing later tokens; the loss mask keeps padding from being graded. They are different masks for different jobs.

loss over real positions loss over all six (3 decimals) targets in a 5-token sequence

To do: Per-position losses [1, 2, 3, 4, 9, 9]; the last two are padding (mask [1,1,1,1,0,0]). The loss over real positions only?
To do: The same losses averaged over ALL six positions, padding included (3 decimals)
To do: A sequence of 5 real tokens has how many next-token targets?

Part 2. Do it in code

shift_targets(tokens, pad_id): inputs = tokens[:-1], targets = tokens[1:]; a target equal to pad_id is replaced by -100 (ignore). Raise ValueError for fewer than 2 tokens.
loss_mask(targets, ignore=-100): a list of 1/0 numbers, 0 where the target is ignored.
masked_mean(losses, mask): sum(loss·mask) / sum(mask). ValueError if the mask sums to 0.
cross_entropy(logits, targets, ignore=-100): logits (n, V), targets length n. Average −log softmax(logits)[target] over positions whose target is not ignore. ValueError if nothing is left.

import numpy as np

def shift_targets(tokens, pad_id):
    pass

def loss_mask(targets, ignore=-100):
    pass

def masked_mean(losses, mask):
    pass

def cross_entropy(logits, targets, ignore=-100):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can loss include padded positions without a defined target policy?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A commonly used rough dense-transformer training estimate is

C\approx6ND,

with N model parameters, D training tokens and C floating-point operations. It omits some architecture-dependent work and is not a hardware throughput guarantee. Effective token batch is approximately $B\times T\times A$ for microbatch B, length T and A accumulation steps.

Worked example

For N=1 million parameters and D=100 million training tokens, the rough compute estimate is 6×10^6×10^8=6×10^14 FLOPs. A microbatch of 4 sequences of length 8 accumulated twice contributes 64 token positions per optimizer update. Neither calculation includes padding exclusions or communication time.

Reference code

The following example uses the Python standard library and does not call a remote service.

N,D=1_000_000,100_000_000
assert 6*N*D == 600_000_000_000_000
batch,length,accumulation=4,8,2
assert batch*length*accumulation == 64

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Compute-optimal ratios are model- and study-dependent. Train/test leakage can arise from shared documents and overlap. Tokens per update and optimizer steps are different units.