L5.6 · LESSON 6 OF 6

Parameter counting

Trace a transformer block, distinguish normalization variants and count a simplified block's parameters.

THE IDEA

Counting parameters

Counting is simple if you state your assumptions and count each distinct matrix once. For the simplified block with d = 4 and f = 8: attention has four d×d matrices (4·16 = 64), the MLP two matrices (2·4·8 = 64), so the block has 128 matrix weights. Biases, normalisation scales, embedding tables and any positional parameters are separate items you add only if the model has them.

width d4hidden f8layers1vocabulary100

biases gated MLP two norm layers per block tied input/output embeddings

part	count
attention (4 matrices of d×d)	64
MLP (2 matrices of d×f)	64
norm scales	0
one block	128
1 block	128
embedding table (shared with the output layer)	400
total (without positional parameters)	528

The sequence length appears nowhere in this table: doubling the context does not add dense weights (it only makes the attention computation and cache larger).

Change the assumptions. The sequence length does not appear.

Predict first. Does doubling context length double the number of learned dense-projection weights?

Chapter context

A transformer combines attention with per-position feed-forward computation and residual paths. Exact ordering and normalization differ between architectures; learn a stated variant instead of assuming every transformer has an identical block.

TRY IT YOURSELF

Part 1. Do it hands-on

Count a block under different assumptions, then write the counting functions and check a GPT-2-small-sized block.

width d4hidden f8layers1vocabulary100

biases gated MLP two norm layers per block tied input/output embeddings

part	count
attention (4 matrices of d×d)	64
MLP (2 matrices of d×f)	64
norm scales	0
one block	128
1 block	128
embedding table (shared with the output layer)	400
total (without positional parameters)	528

The sequence length appears nowhere in this table: doubling the context does not add dense weights (it only makes the attention computation and cache larger).

Pen-and-paper values for d = 4, f = 8:

attention weights block (attention + MLP) block with biases and two norms vocab 100, d 4, tied: embedding weights

To do: d = 4. Attention has four d×d matrices (Q, K, V, output) and no biases. How many weights?
To do: Add the two MLP matrices (f = 8). The simplified block's matrix weights in total?
To do: Now count biases too (4d in attention, f + d in the MLP) and two normalisation layers with d scale values each. Total for the block?
To do: A vocabulary of 100 and d = 4. The input embedding table has 400 weights. If the output layer reuses the same table (tied), how many weights do the two together count?

Part 2. Do it in code

attn_params(d, bias=False): four d×d matrices (+ 4d biases if bias).
mlp_params(d, f, gated=False, bias=False): 2·d·f or 3·d·f weights. Biases: f + d for the simple MLP, 2f + d for the gated one.
block_params(d, f, gated=False, bias=False, norms=0): attention + MLP + norms·d (each norm has d scale values).
model_params(layers, d, f, vocab, tied=True, gated=False, bias=False, norms_per_block=2, final_norm=True): the embedding table (vocab·d), plus an untied output matrix (vocab·d) only when tied is False, plus layers blocks, plus one final norm (d) when final_norm.

def attn_params(d, bias=False):
    pass

def mlp_params(d, f, gated=False, bias=False):
    pass

def block_params(d, f, gated=False, bias=False, norms=0):
    pass

def model_params(layers, d, f, vocab, tied=True, gated=False, bias=False, norms_per_block=2, final_norm=True):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does doubling context length double the number of learned dense-projection weights?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One possible pre-norm block is

h=x+\operatorname{Attention}(\operatorname{Norm}(x)),\quad y=h+\operatorname{MLP}(\operatorname{Norm}(h)).

For standard full-width Q/K/V/O projections and an ungated d→f→d MLP, matrix weights total $4d^2+2df$ , excluding bias and normalization. GQA changes K/V projection dimensions.

For one feature vector x of width d, LayerNorm uses $\mu=\frac1d\sum_i x_i$ and $v=\frac1d\sum_i(x_i-\mu)^2$ , then normalizes $(x_i-\mu)/\sqrt{v+\epsilon}$ before its learned affine transform. RMSNorm instead scales using $\sqrt{\frac1d\sum_i x_i^2+\epsilon}$ without the same mean subtraction. Neither definition normalizes across arbitrary axes unless those axes are part of the specified layer.

Worked example

Let d=4 and f=8. Four attention matrices contain 4×4²=64 weights. The two MLP matrices contain 2×4×8=64. The simplified block has 128 matrix weights. Embedding tables, learned normalization scales, biases and any positional parameters are outside this count.

For x=[3,4] without affine transforms or epsilon, LayerNorm produces [-1,1], while RMSNorm divides by sqrt(12.5). The outputs differ because RMSNorm does not center the vector.

Reference code

The following example uses the Python standard library and does not call a remote service.

width,hidden=4,8
attention=4*width*width
mlp=2*width*hidden
assert attention == 64 and mlp == 64
assert attention+mlp == 128
query_heads,kv_heads=4,2
assert query_heads//kv_heads == 2
from math import sqrt,isclose
x=[3.,4.]
mean=sum(x)/len(x)
var=sum((v-mean)**2 for v in x)/len(x)
assert [(v-mean)/sqrt(var) for v in x] == [-1.,1.]
rms=sqrt(sum(v*v for v in x)/len(x))
assert isclose(sum((v/rms)**2 for v in x)/len(x),1)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Parameter-count formulas depend on the architecture. RMSNorm and LayerNorm do not perform identical transformations. Smaller KV caches do not mean the model has no attention or no queries.