D1.3 · LESSON 3 OF 6

Layers and shapes

Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.

THE IDEA

A layer is a matrix

A dense layer connects every input to every output unit. With in inputs and out units it has a weight matrix W of shape (in, out) and a bias vector b of shape (out,). The layer computes x @ W + b.

Real data comes in batches: many examples at once, stacked as rows. A batch has shape (batch, features). The batch axis is not part of the layer: the same W is applied to every row, so the output is (batch, out) for any batch size.

the shape rule

(batch × in) @ (in × out)  →  (batch × out)

batch size

inputshidden 1hidden 2outputs

layer	x shape	W shape	b shape	output (x @ W + b)	parameters
1	(32, 6)	(6, 5)	(5,)	(32, 5)	35
2	(32, 5)	(5, 4)	(4,)	(32, 4)	24
3	(32, 4)	(4, 3)	(3,)	(32, 3)	15
total parameters					74

rule: (batch × in) @ (in × out) → (batch × out) the parameter count does not depend on the batch size

Change the batch size and the layer widths. Every shape updates, and the parameter count ignores the batch.

Predict first. A batch of five examples with 3 features each goes through a layer of width 4. What shape comes out?

Where shape bugs hide

Chapter context

A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.

TRY IT YOURSELF

Part 1. Do it hands-on

Configure the network so every shape matches what the tasks demand.

batch size

inputshidden 1outputs

layer	x shape	W shape	b shape	output (x @ W + b)	parameters
1	(1, 2)	(2, 3)	(3,)	(1, 3)	9
2	(1, 3)	(3, 1)	(1,)	(1, 1)	4
total parameters					13

rule: (batch × in) @ (in × out) → (batch × out) the parameter count does not depend on the batch size

To do: Feed batches of 32 examples that each have 6 features (batch 32, input width 6)
To do: Make the first weight matrix W1 have shape (6, 5)
To do: Use exactly two hidden layers, and make the second weight matrix W2 have shape (5, 4)
To do: Make the network return one row of 3 scores per example: output shape (32, 3)

Part 2. Do it in code

init_layer(n_in, n_out, seed=0): return (W, b) with W of shape (n_in, n_out) drawn from a normal distribution with standard deviation sqrt(2 / n_in) using np.random.default_rng(seed), and b zeros of shape (n_out,). Raise ValueError if n_in or n_out is less than 1.
dense(X, W, b): X has shape (batch, n_in). Return X @ W + b with shape (batch, n_out). If X is not 2-D or its feature count does not match W, raise ValueError.
layer_shapes(sizes): for sizes like [4, 8, 3] return a list with one (W shape, b shape) pair per layer.

import numpy as np

def init_layer(n_in, n_out, seed=0):
    # W: (n_in, n_out), normal with std sqrt(2 / n_in); b: zeros (n_out,)
    pass

def dense(X, W, b):
    # (batch, n_in) @ (n_in, n_out) + (n_out,) -> (batch, n_out)
    pass

def layer_shapes(sizes):
    # [4, 8, 3] -> [((4, 8), (8,)), ((8, 3), (3,))]
    pass

print(layer_shapes([4, 8, 3]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

A batch of five 3D inputs passes through a width-four layer. What is the output shape?

A(3, 4): the batch axis disappears after the layer.B(5, 3): the width of the layer is ignored.C(5, 4): one row per example and one column per output unit.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

z_j=\sum_iw_{ji}x_i+b_j,\quad h_j=\phi(z_j),\quad \operatorname{ReLU}(z)=\max(0,z).

A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.

Worked example

For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.