A baseline
Assemble a tiny causal language-model experiment with a clear data split, baseline, training loop and evaluation.
THE IDEA
Always beat a baseline first
Before any transformer, fit something simple: a uniform guess (loss ln V) and a smoothed bigram model (the counts from the language-modelling chapter). They cost almost nothing, give you numbers to beat, and show what samples from a weak model look like.
| model | held-out loss | perplexity | |
|---|---|---|---|
| uniform guess | 0.693 | 2 | |
| smoothed bigram (add-one) | 0.142 | 1.15 |
Any new model must beat these numbers on the same held-out text, otherwise it has learned nothing useful. Try a held-out text with different habits from the training text.
Predict first. Should the baseline and the new model be evaluated on unrelated test text?
Chapter context
The capstone joins the earlier pieces into one inspectable experiment. The goal is to understand the model and its evidence, not to claim that a toy trained on a few paragraphs competes with a production assistant.
TRY IT YOURSELF
Part 1. Do it hands-on
Compute the baseline numbers by hand, then build the count-based baseline, its evaluation and a sampler.
| model | held-out loss | perplexity | |
|---|---|---|---|
| uniform guess | 0.693 | 2 | |
| smoothed bigram (add-one) | 0.142 | 1.15 |
Any new model must beat these numbers on the same held-out text, otherwise it has learned nothing useful. Try a held-out text with different habits from the training text.
- To do: The dumbest baseline gives every one of V = 27 characters the same probability. Its loss is ln 27 (3 decimals)
- To do: Its perplexity (exp of the loss)?
- To do: A smoothed bigram model (add-one, V = 2) trained on "ababa" gives P(next | current) = 0.75 for every transition in that text. Its loss there (3 decimals)
Part 2. Do it in code
- bigram_counts(tokens, V): (V, V) counts of how often token j follows token i.
- bigram_probs(counts, alpha=1.0): additive smoothing row by row, (count + alpha) / (row sum + alpha·V).
- eval_loss(P, tokens): the average −ln P[current, next] over the adjacent pairs of tokens.
- uniform_loss(V): the loss of a model that gives every token probability 1/V.
- perplexity(loss): exp(loss).
- sample_bigram(P, start, n, seed): n tokens generated by drawing from P rows (np.random.default_rng(seed).choice), starting from `start` (not included in the output).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Should the baseline and new model use unrelated test text?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
For a token sequence of length n and context length T, stride-one windows containing T inputs and their shifted targets require T+1 original tokens. The number of complete windows is max(0,n-T). Keep this counting convention distinct from nonoverlapping batching.
Worked example
A 400-token text is split into 320 training tokens and 80 validation tokens before windowing. With T=8, the training part contains 320−8=312 complete shifted windows, while validation contains 72. Constructing windows before the split would risk overlap across the boundary.
Reference code
The following example uses the Python standard library and does not call a remote service.
text='abcd'*100
train,validation=text[:320],text[320:]
context=8
windows=[train[i:i+context+1] for i in range(len(train)-context)]
assert len(text) == 400
assert len(windows) == 312
assert all(len(w)==9 for w in windows)
assert len(validation)-context == 72
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Toy output quality is not a benchmark claim. Splitting overlapping windows after construction leaks shared text. Save the tokenizer and architecture configuration, not only raw weights.