L10.5 · LESSON 5 OF 6

Sampling and evaluation

Assemble a tiny causal language-model experiment with a clear data split, baseline, training loop and evaluation.

THE IDEA

Judging the result honestly

Report held-out loss (and its perplexity) next to training loss, compare with the baseline, and generate samples with a stated decoding configuration (temperature, seed). Samples are random draws, so one checkpoint gives different text for different seeds; that is not a bug. Show some bad samples too, not only the best-looking one.

training loss1.2validation loss2

gap = 2 − 1.2 = 0.8 validation perplexity = exp(2) = 7.389 uniform baseline loss ln 27 = 3.296 verdict: overfitting: training loss is much lower than held-out loss

sampling seed1

samples from the same model with seed 1: baabbbaabaababbbabbbbaab Different seeds give different samples from one checkpoint; report the decoding settings and seed, and show weak samples as well as good ones.

Set the two losses to see the gap and a verdict. Then change the sampling seed.

Predict first. Must the same checkpoint produce identical samples under different seeds?

Chapter context

The capstone joins the earlier pieces into one inspectable experiment. The goal is to understand the model and its evidence, not to claim that a toy trained on a few paragraphs competes with a production assistant.

TRY IT YOURSELF

Part 1. Do it hands-on

Read losses and perplexity by hand, then write the sampler with temperature and a small diagnosis function.

training loss1.2validation loss2

gap = 2 − 1.2 = 0.8 validation perplexity = exp(2) = 7.389 uniform baseline loss ln 27 = 3.296 verdict: overfitting: training loss is much lower than held-out loss

sampling seed1

gap for 1.2 and 2.0 perplexity exp(2.0) better than the ln 27 baseline? (yes/no)

To do: Training loss 1.2, held-out (validation) loss 2.0. The generalisation gap (validation − training)?
To do: The validation perplexity exp(2.0) (3 decimals)
To do: The 27-character uniform baseline has validation loss ln 27. Is a model with validation loss 2.0 better than that baseline? Type yes or no

Part 2. Do it in code

perplexity(loss) = exp(loss). gap(train_loss, val_loss) = val − train.
sample_text(P, itos, start, n, seed, temperature=1.0): from the bigram probability matrix P, generate n characters starting from token id `start` (not included). Apply temperature to each row by raising probabilities to the power 1/T and renormalising (T must be positive, ValueError otherwise). Use np.random.default_rng(seed). Return the string.
distinct_samples(P, itos, start, n, seeds, temperature=1.0): the number of different strings produced by the given seeds.
diagnose(train_loss, val_loss, baseline_loss): return one of "no better than baseline" (val_loss >= baseline_loss), "overfitting" (gap above 0.5 and val_loss below baseline), or "healthy".

import math
import numpy as np

def perplexity(loss):
    pass

def gap(train_loss, val_loss):
    pass

def sample_text(P, itos, start, n, seed, temperature=1.0):
    pass

def distinct_samples(P, itos, start, n, seeds, temperature=1.0):
    pass

def diagnose(train_loss, val_loss, baseline_loss):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must the same checkpoint produce identical samples under different seeds?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a token sequence of length n and context length T, stride-one windows containing T inputs and their shifted targets require T+1 original tokens. The number of complete windows is max(0,n-T). Keep this counting convention distinct from nonoverlapping batching.

Worked example

A 400-token text is split into 320 training tokens and 80 validation tokens before windowing. With T=8, the training part contains 320−8=312 complete shifted windows, while validation contains 72. Constructing windows before the split would risk overlap across the boundary.

Reference code

The following example uses the Python standard library and does not call a remote service.

text='abcd'*100
train,validation=text[:320],text[320:]
context=8
windows=[train[i:i+context+1] for i in range(len(train)-context)]
assert len(text) == 400
assert len(windows) == 312
assert all(len(w)==9 for w in windows)
assert len(validation)-context == 72

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Toy output quality is not a benchmark claim. Splitting overlapping windows after construction leaks shared text. Save the tokenizer and architecture configuration, not only raw weights.