D1.4 · LESSON 4 OF 6

A forward pass

Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.

THE IDEA

Computing a network step by step

A forward pass takes an input through the layers in order. Each layer needs the previous layer's output as its input, so nothing can be skipped or reordered: layer 2 has to wait for layer 1.

The values computed along the way are worth keeping. The score z before the activation and the activation a after it are exactly what the next chapter reuses to compute gradients. A forward value is a number the network produced, not a derivative.

x11x22

showing input only

x = [1, 2] layer 1: not computed yet layer 2: not computed yet

A 2→3→1 network. Choose an input and press next layer to see each matrix product, the ReLU, and the output.

Predict first. With the default input x = [1, 2], which hidden units does ReLU switch off (output 0)?

Chapter context

A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.

TRY IT YOURSELF

Part 1. Do it hands-on

Run the same network by hand for the input x = [−1, 0.5] and enter every intermediate value.

x = [-1, 0.5] W1 = [[0.5,-1,2],[1,0.5,-0.5]] b1 = [0.5, -0.5, 0.5] W2 = [[1],[2],[-1]] b2 = [0.25]

z1 =

h = relu(z1) =

z2 =

To do: Compute the three pre-activations z1 of the hidden layer for the new input x = [-1, 0.5]
To do: Apply ReLU to get the hidden activations h
To do: Compute the output score z2 = h @ W2 + b2

Part 2. Do it in code

forward(X, layers): layers is a list of (W, b) pairs. Apply every layer in order: z = a @ W + b. Hidden layers apply ReLU, the last layer stays linear.
Return (out, cache) where out has shape (batch, last width) and cache is a list with one (z, a) pair per layer (its pre-activation and activation).
Raise ValueError if layers is empty or if the shapes of two consecutive layers do not fit together.

import numpy as np

def forward(X, layers):
    # layers: [(W1, b1), (W2, b2), ...]; ReLU after every layer except the last
    # return (out, cache) with cache = [(z1, a1), (z2, a2), ...]
    pass

W1 = np.array([[0.5, -1, 2], [1, 0.5, -0.5]]); b1 = np.array([0.5, -0.5, 0.5])
W2 = np.array([[1], [2], [-1]]); b2 = np.array([0.25])
out, cache = forward(np.array([[1.0, 2.0]]), [(W1, b1), (W2, b2)])
print(out)

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must the second layer wait for the first layer's outputs?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

z_j=\sum_iw_{ji}x_i+b_j,\quad h_j=\phi(z_j),\quad \operatorname{ReLU}(z)=\max(0,z).

A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.

Worked example

For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.