Training data
Describe a pretraining loop and reason about data, batch size, precision and a simplified compute budget.
THE IDEA
Data decides what can be learned, and what can be trusted
Pretraining reads huge amounts of text. Three habits keep that honest. Deduplicate: repeated documents waste compute and can be memorised. Cut text into fixed-length windows the model can process. And hold out a validation set that the model never trains on, so its score means something.
The order matters. Windows from the same document usually overlap or sit next to each other. If you cut windows first and then split them, neighbouring windows land on both sides and the validation text is partly in training.
One 10-token document. Every third window is sent to validation (orange), the rest to training (blue).
Predict first. Why split documents before creating neighbouring windows?
Chapter context
Pretraining optimizes next-token predictions over a dataset. Its costs depend on the architecture, sequence lengths, implementation and hardware. A useful budget model states which work it approximates and which work it ignores.
TRY IT YOURSELF
Part 1. Do it hands-on
Count windows, find the overlap that causes leakage, then write deduplication, windowing and a leak-free split.
One 10-token document. Every third window is sent to validation (orange), the rest to training (blue).
- To do: A document has 10 tokens. Windows of size 4 with stride 2 start at 0, 2, 4, …. How many full windows fit?
- To do: Two neighbouring windows (positions 0–3 and 2–5) overlap. How many token positions do they share? If one went to training and the other to validation, that many tokens would leak
- To do: Documents [a, b, a, c, b]. Removing exact duplicates (keep the first copy) removes how many documents?
Part 2. Do it in code
- dedup(docs): remove exact duplicates, keeping the first occurrence and the original order.
- make_windows(n, size, stride): list of (start, end) for every full window of a length-n document. ValueError for size < 1 or stride < 1.
- split_docs(docs, val_frac, seed): shuffle the document indices with np.random.default_rng(seed).permutation, put the first max(1, round(val_frac·n)) into validation. Return (train_docs, val_docs) as lists of documents.
- leaks(train_windows, val_windows): windows are (doc_id, start, end). True if a train window and a validation window come from the same doc_id and their ranges overlap.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Why split documents before creating neighboring windows?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A commonly used rough dense-transformer training estimate is
with N model parameters, D training tokens and C floating-point operations. It omits some architecture-dependent work and is not a hardware throughput guarantee. Effective token batch is approximately for microbatch B, length T and A accumulation steps.
Worked example
For N=1 million parameters and D=100 million training tokens, the rough compute estimate is 6×10^6×10^8=6×10^14 FLOPs. A microbatch of 4 sequences of length 8 accumulated twice contributes 64 token positions per optimizer update. Neither calculation includes padding exclusions or communication time.
Reference code
The following example uses the Python standard library and does not call a remote service.
N,D=1_000_000,100_000_000
assert 6*N*D == 600_000_000_000_000
batch,length,accumulation=4,8,2
assert batch*length*accumulation == 64
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Compute-optimal ratios are model- and study-dependent. Train/test leakage can arise from shared documents and overlap. Tokens per update and optimizer steps are different units.