D1.2 · LESSON 2 OF 6

Activations

Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.

THE IDEA

Why stacking linear layers is not enough

A layer of neurons with no activation is just a matrix multiplication. Two of them in a row are two matrix multiplications, and the product of two matrices is itself one matrix. A hundred linear layers can always be replaced by a single one, so depth would buy nothing.

A nonlinear activation between the layers breaks that collapse. Each neuron applies a bent function to its score, and a network of bent functions can draw curved boundaries that no single line can.

Predict first. Two layers with no activation: x @ W1 @ W2, where W1 and W2 are both 2×2. How many 2×2 matrices does it take to do exactly the same job?

The activations in common use

activation f(z)slope f'(z)

Sigmoid(z) = 1 / (1 + e^-z) f(1) = 0.7311 slope = 0.1966 range: (0, 1) Squashes any score into (0, 1). The slope peaks at 0.25 when z = 0 and almost vanishes for |z| above about 5.

Pick an activation and slide the probe. The solid curve is the activation, the dashed one is its slope. The slope is what learning uses.

Two things to notice. ReLU returns zero for any negative score, so ReLU of −1 is 0, and has slope exactly 1 for positive scores. Sigmoid squashes into (0, 1) but is nearly flat for large |z|, so a neuron whose score is very large or very small has a slope close to zero and learns very slowly (it saturates).

Chapter context

A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.

TRY IT YOURSELF

Part 1. Do it hands-on

Multiply the two layer matrices yourself, then use the same two layers with a ReLU between them to find an input where the collapse fails.

W1 = [[1, -1], [2, 0.5]] W2 = [[0.5, 1], [-1, 2]]

C = W1 @ W2 = [ ; ]

Input x = [ , ]

no activation: x @ W1 @ W2 = [0, 0] with ReLU between: relu(x @ W1) @ W2 = [0, 0] biggest difference = 0

probe z0

sigmoid(0) = 0.5 slope = 0.25

true C is hidden until you commit: check it against your own multiplication (2×2 matrix).

To do: Enter the single 2x2 matrix C equal to W1 @ W2, so that x @ C gives the same as passing x through both layers
To do: With a ReLU between the layers, find an input x where the network and the collapsed matrix C differ by at least 1 in some output
To do: Move the sigmoid probe to a score z where the sigmoid slope is below 0.01 (the unit has all but stopped learning)

Part 2. Do it in code

sigmoid(z): 1 / (1 + e^-z) for a scalar or array. It must be safe for very large positive and negative z: sigmoid(-1000) is 0.0 and sigmoid(1000) is 1.0, with no overflow warning.
relu(z) and relu_grad(z): max(0, z), and its slope (1 where z > 0, otherwise 0), for arrays.
leaky_relu(z, alpha=0.01): z where z > 0, otherwise alpha * z.
collapse(W1, b1, W2, b2): the two linear layers x @ W1 + b1 then @ W2 + b2 equal one layer x @ W + b. Return (W, b).

import numpy as np

def sigmoid(z):
    pass

def relu(z):
    pass

def relu_grad(z):
    pass

def leaky_relu(z, alpha=0.01):
    pass

def collapse(W1, b1, W2, b2):
    # return (W, b) with x @ W + b == (x @ W1 + b1) @ W2 + b2
    pass

print(relu(np.array([-1.0, 2.0])))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What does ReLU do to -1?

AIt returns zero, because ReLU outputs max(0, z).BIt returns −1, because ReLU leaves its input unchanged.CIt returns 1, because ReLU takes the absolute value.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

z_j=\sum_iw_{ji}x_i+b_j,\quad h_j=\phi(z_j),\quad \operatorname{ReLU}(z)=\max(0,z).

A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.

Worked example

For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.