Capacity and parameters
Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.
THE IDEA
Counting parameters
Every dense layer has in × out weights and out biases. Biases are parameters too. A layer from 3 inputs to 4 outputs has 3×4 + 4 = 16 parameters. A 3→4→2 network has 16 + (4×2 + 2) = 26. Add up the layers; the batch size never enters.
Predict first. How many parameters does a dense layer from 3 inputs to 4 outputs have, with bias?
More parameters, more capacity, but not a guarantee
A wider layer can represent more complicated functions. The data below is two rings: one class inside, one outside. No straight line separates them. Each network was trained the same way (same seed, 1,500 full-batch gradient steps). Pick a width and a training stage and look at the regions it learned.
| width | params | final accuracy |
|---|---|---|
| 1 | 5 | 75% |
| 2 | 9 | 97% |
| 3 | 13 | 98% |
| 4 | 17 | 100% |
| 8 | 33 | 100% |
| 16 | 65 | 100% |
Chapter context
A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.
TRY IT YOURSELF
Part 1. Do it hands-on
Design a network to an exact parameter budget, and find the smallest network that still separates the rings.
| layer | weights | biases | total |
|---|---|---|---|
| 1 | 8 | 4 | 12 |
| 2 | 4 | 1 | 5 |
| all | 17 (target 100) | ||
| width | params | final accuracy |
|---|---|---|
| 1 | 5 | 75% |
| 2 | 9 | 97% |
| 3 | 13 | 98% |
| 4 | 17 | 100% |
| 8 | 33 | 100% |
| 16 | 65 | 100% |
- To do: Design a network with 2 inputs, 1 output and one or two hidden layers whose total parameter count is exactly 100
- To do: On the circles data, find a one-hidden-layer network with at most 10 parameters that classifies at least 95% of the training points correctly
Part 2. Do it in code
- layer_params(n_in, n_out): weights plus biases of one dense layer. Raise ValueError if either width is less than 1.
- count_params(sizes): total for a network with widths like [2, 8, 1]. Raise ValueError for fewer than two sizes.
- params_by_layer(sizes): list of per-layer counts.
- count_arrays(layers): given a list of (W, b) NumPy pairs, count the numbers actually stored (W.size + b.size for each).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
How many parameters are in a dense layer from three inputs to four outputs with bias?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.
Worked example
For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.
Reference code
The following example uses the Python standard library and does not call a remote service.
x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.