L6.5 · LESSON 5 OF 6

Scaling and budgets

Describe a pretraining loop and reason about data, batch size, precision and a simplified compute budget.

THE IDEA

Budgets: parameters, tokens and compute

A rough estimate for the compute of training a dense transformer is C ≈ 6 · N · D floating-point operations, where N is the number of parameters and D the number of training tokens. The worked example: N = 10⁶ and D = 10⁸ give 6·10¹⁴ FLOPs. If a machine really sustains 10¹² FLOP/s on the job, that is 600 seconds.

For a fixed C the formula shows the trade-off: a model twice as big can see only half as many tokens. Published scaling-law studies fit how loss varies with N, D and C, and use such fits to decide the split.

log10 of compute budget (FLOPs)10^14.8 = 6.03e+14

parameters N	tokens D = C / 6N	tokens per parameter	seconds at 10¹² FLOP/s
1e+5	1.00e+9	10042.66	603
1e+6	1.00e+8	100.427	603
1e+7	1.00e+7	1.004	603
1e+8	1.00e+6	0.01	603

At a fixed budget, a bigger model means fewer tokens. Empirical "scaling laws" describe how loss tends to change with N, D and C under the conditions studied, and help to choose a split. They are fits to experiments, not guarantees for every setting.

Choose a budget. Each row is a different split of the same compute.

Predict first. Does more model size always beat more data under every fixed budget?

Chapter context

Pretraining optimizes next-token predictions over a dataset. Its costs depend on the architecture, sequence lengths, implementation and hardware. A useful budget model states which work it approximates and which work it ignores.

TRY IT YOURSELF

Part 1. Do it hands-on

Do the compute arithmetic by hand, then code the budget calculators.

log10 of compute budget (FLOPs)10^14.8 = 6.03e+14

parameters N	tokens D = C / 6N	tokens per parameter	seconds at 10¹² FLOP/s
1e+5	1.00e+9	10042.66	603
1e+6	1.00e+8	100.427	603
1e+7	1.00e+7	1.004	603
1e+8	1.00e+6	0.01	603

6·N·D for N = 10⁶, D = 10⁸ tokens for the same budget with N = 2·10⁶ seconds for 6·10¹⁴ at 10¹² FLOP/s

To do: Rule of thumb: training compute ≈ 6 · N · D FLOPs. N = 1,000,000 parameters, D = 100,000,000 tokens. Type the FLOPs (e.g. 6e14)
To do: Same budget 6·10¹⁴ FLOPs but the model has N = 2,000,000 parameters. How many tokens D can you afford? Type it
To do: A machine sustains 10¹² FLOP/s on this job. Seconds for 6·10¹⁴ FLOPs?

Part 2. Do it in code

train_flops(N, D): 6·N·D.
tokens_for_budget(C, N): D = C / (6N). N must be positive (ValueError).
params_for_budget(C, D): N = C / (6D).
training_seconds(C, flops_per_sec, utilization=1.0): C / (flops_per_sec · utilization). utilization must be in (0, 1] (ValueError).
split_budget(C, ratio): choose N and D so that 6·N·D = C and D = ratio·N. Return (N, D).

def train_flops(N, D):
    pass

def tokens_for_budget(C, N):
    pass

def params_for_budget(C, D):
    pass

def training_seconds(C, flops_per_sec, utilization=1.0):
    pass

def split_budget(C, ratio):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does more model size always beat more data under every fixed budget?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A commonly used rough dense-transformer training estimate is

C\approx6ND,

with N model parameters, D training tokens and C floating-point operations. It omits some architecture-dependent work and is not a hardware throughput guarantee. Effective token batch is approximately $B\times T\times A$ for microbatch B, length T and A accumulation steps.

Worked example

For N=1 million parameters and D=100 million training tokens, the rough compute estimate is 6×10^6×10^8=6×10^14 FLOPs. A microbatch of 4 sequences of length 8 accumulated twice contributes 64 token positions per optimizer update. Neither calculation includes padding exclusions or communication time.

Reference code

The following example uses the Python standard library and does not call a remote service.

N,D=1_000_000,100_000_000
assert 6*N*D == 600_000_000_000_000
batch,length,accumulation=4,8,2
assert batch*length*accumulation == 64

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Compute-optimal ratios are model- and study-dependent. Train/test leakage can arise from shared documents and overlap. Tokens per update and optimizer steps are different units.