D1.6 · LESSON 6 OF 6

Classification by a network

Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.

THE IDEA

What a network outputs: logits

The last layer produces raw scores called logits. They are not probabilities: they can be negative, can be larger than 1, and do not add up to anything in particular. To classify, add an output rule that matches the task: a sigmoid for one yes/no probability, a softmax to turn several scores into probabilities that sum to 1, or just take the largest logit as the predicted class.

Predict first. Are the raw logits of a classifier guaranteed to sum to one?

Hidden layers make XOR possible

XOR outputs 1 when exactly one input is on. A single neuron cannot compute it, because no straight line separates {(0,1), (1,0)} from {(0,0), (1,1)}. A hidden layer fixes this: one hidden unit can compute OR, another AND, and the output unit fires when OR is on and AND is off.

Searched every (w1, w2, b) on a grid from -4 to 4 in steps of 0.5 (17³ = 4913 settings): none makes a single step neuron output 0, 1, 1, 0.

Brute force over a weight grid: a single neuron cannot do it.

hidden width:

epoch4

dataset: Three blobs (60 points) network: 2 → 4 → 3 (27 parameters) after 1500 epochs: loss 0.001 accuracy 100%

width	params	final accuracy
1	5	98%
2	9	100%
3	13	100%
4	17	100%
8	33	100%
16	65	100%

Three classes with a softmax output. Select a width and watch the regions form as training proceeds.

Choosing the output rule

Match the output rule to the loss: pair a sigmoid with binary cross-entropy and a softmax with categorical cross-entropy, as the loss expects.

Don't

Apply a softmax and then feed the probabilities into a loss that applies its own softmax, which silently squashes the learning signal.

Chapter context

A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.

TRY IT YOURSELF

Part 1. Do it hands-on

Set nine numbers so a 2-2-1 network of step neurons computes XOR.

w110w120w210w220b10b20v10v20c0

x1	x2	XOR wants
0	0	0
0	1	1
1	0	1
1	1	0

hidden unit j: step(w1j·x1 + w2j·x2 + bj) (unit 1 uses w11, w21, b1; unit 2 uses w12, w22, b2) output: step(v1·h1 + v2·h2 + c)

To do: Set the nine numbers so the 2-2-1 step network outputs XOR: 0, 1, 1, 0 for the four inputs
To do: Make the two hidden units compute different things (their four outputs must not be identical)

Part 2. Do it in code

softmax(z): turn logits into probabilities along the last axis, for a vector or for a (batch, classes) matrix. It must not overflow for large logits: softmax([1000, 1000, 1000]) is one third each.
predict(logits): the index of the largest logit in each row, shape (batch,).
cross_entropy(logits, y): mean over the batch of -log(probability of the true class y_i), computed without ever taking log(0). Use the log-sum-exp form.

import numpy as np

def softmax(z):
    pass

def predict(logits):
    pass

def cross_entropy(logits, y):
    pass

print(softmax(np.array([1.0, 2.0, 3.0])))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Are raw logits guaranteed to sum to one?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

z_j=\sum_iw_{ji}x_i+b_j,\quad h_j=\phi(z_j),\quad \operatorname{ReLU}(z)=\max(0,z).

A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.

Worked example

For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.