An experiment report
Assemble a tiny causal language-model experiment with a clear data split, baseline, training loop and evaluation.
THE IDEA
The report
Finish by writing down what you did so someone else (or future you) can check it: the architecture, the token budget, how the data was split, the baseline, the results, the failure cases and how to reproduce the run. Export the model and tokenizer state with the instructions.
A worked count: a 400-token text split 320/80 gives 312 training and 72 validation windows at T = 8. Building windows before the split would give 392 windows, some of which cross the boundary.
A report also exports the model, the tokenizer/vocabulary and the instructions (including the seed) to repeat the experiment. If generation is still poor, say so and show the failing samples.
Predict first. What should the report say if generation is still poor?
Chapter context
The capstone joins the earlier pieces into one inspectable experiment. The goal is to understand the model and its evidence, not to claim that a toy trained on a few paragraphs competes with a production assistant.
TRY IT YOURSELF
Part 1. Do it hands-on
Work out the budget and find the missing sections, then write the report generator, a section checker and a reproducibility test.
A report also exports the model, the tokenizer/vocabulary and the instructions (including the seed) to repeat the experiment. If generation is still poor, say so and show the failing samples.
- To do: Token budget = steps × batch × T = 200 × 16 × 8. Type it
- To do: The training part has 4,000 tokens. How many passes (epochs) over it does that budget make?
- To do: A report has the sections Architecture, Token budget, Data split, Results and Reproduce. The seven required sections also include Baseline and Failure cases. How many are missing?
Part 2. Do it in code
- token_budget(steps, batch, T) = steps · batch · T. epochs(budget, train_tokens) = budget / train_tokens (ValueError if train_tokens < 1).
- REQUIRED is the list of section titles a report must contain: Architecture, Token budget, Data split, Baseline, Results, Failure cases, Reproduce.
- make_report(config, metrics, failures): return Markdown text with a "## " heading for every required section. Architecture lists the config; Token budget shows the computed budget; Baseline and Results show metrics (train_loss, val_loss, baseline_loss); Failure cases lists every item of `failures` (or the line "none observed" if empty); Reproduce shows the seed.
- missing_sections(text): the required section titles that do not appear as a "## <title>" heading, in the REQUIRED order.
- reproducible(run, seed): run(seed) twice; True if both results are equal.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
What should a report say if generation is still poor?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
For a token sequence of length n and context length T, stride-one windows containing T inputs and their shifted targets require T+1 original tokens. The number of complete windows is max(0,n-T). Keep this counting convention distinct from nonoverlapping batching.
Worked example
A 400-token text is split into 320 training tokens and 80 validation tokens before windowing. With T=8, the training part contains 320−8=312 complete shifted windows, while validation contains 72. Constructing windows before the split would risk overlap across the boundary.
Reference code
The following example uses the Python standard library and does not call a remote service.
text='abcd'*100
train,validation=text[:320],text[320:]
context=8
windows=[train[i:i+context+1] for i in range(len(train)-context)]
assert len(text) == 400
assert len(windows) == 312
assert all(len(w)==9 for w in windows)
assert len(validation)-context == 72
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Toy output quality is not a benchmark claim. Splitting overlapping windows after construction leaks shared text. Save the tokenizer and architecture configuration, not only raw weights.