D1.1 · LESSON 1 OF 6

Weighted sums

Compute a neural-network forward pass and count the parameters of a small multilayer perceptron.

THE IDEA

One neuron is a weighted vote

A neuron takes some numbers in, multiplies each by its own weight, adds them up, and adds one more number, the bias. The result is a single score, usually written z:

the weighted sum

z = w1·x1 + w2·x2 + ... + b        which is the dot product  w · x  plus  b

Each weight belongs to one input: a large positive weight means "this input pushes the score up", a negative one means "it pushes the score down". The bias does not depend on the input at all. It shifts the score so the neuron can be more or less willing to fire when every input is zero.

Worked example: with x = [1, 2], w = [0.5, −1] and b = 0.5, the score is 0.5·1 + (−1)·2 + 0.5 = −1.

x11x22w10.5w2-1b0.5

z = w1·x1 + w2·x2 + b = 0.5·1 + -1·2 + 0.5 = -1 a = ReLU(z) = 0

Move the sliders. The orange line is every input where z = 0: on one side the score is positive, on the other negative. Weights tilt the line; the bias slides it.

Predict first. Keep the weights fixed and increase only the bias. What happens to the orange line?

What one neuron can decide

Pass the score through a step (1 if positive, otherwise 0) and the neuron becomes a yes/no decision: which side of the line is the input on? With inputs that are 0 or 1 this is a logic gate. AND needs both inputs on to cross zero, OR needs only one, and the bias is what separates them.

Chapter context

A neuron computes a weighted sum followed by an activation. Layers combine these operations into functions with learnable parameters. The network is a computation to inspect before it is a framework object to instantiate.

TRY IT YOURSELF

Part 1. Do it hands-on

Set the three numbers so one step neuron computes each gate. The table shows the output for all four inputs.

w10w20b0

x1	x2	z	step(z)
0	0	0	0
0	1	0	0
1	0	0	0
1	1	0	0

To do: Make the neuron compute AND: output 1 only for input (1, 1)
To do: Make it compute OR: output 1 unless the input is (0, 0)
To do: Make it compute NAND: output 0 only for input (1, 1)

Part 2. Do it in code

weighted_sum(x, w, b): return z = w . x + b for one input vector. Raise ValueError if x and w have different lengths.
neuron_batch(X, w, b, f=...): X has shape (n, d). Return the n activations f(X @ w + b) as a vector of shape (n,). The default f is the identity.
contributions(x, w): return the vector of products w_i * x_i, so the learner can see which input pushes z up or down.

import numpy as np

def weighted_sum(x, w, b):
    # z = w . x + b ; ValueError if the lengths differ
    pass

def neuron_batch(X, w, b, f=lambda z: z):
    # f(X @ w + b), shape (n,)
    pass

def contributions(x, w):
    # elementwise products w_i * x_i
    pass

print(weighted_sum([1, 2], [0.5, -1], 0.5))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For x=[1,2],w=[.5,-1],b=.5, find the preactivation.

A0, because 0.5 + (−1) + 0.5 = 0 (the weights are added, not multiplied by the inputs).B−1, because 0.5·1 + (−1)·2 + 0.5 = −1.C−1.5, because 0.5·1 + (−1)·2 = −1.5 and the bias is not part of z.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

z_j=\sum_iw_{ji}x_i+b_j,\quad h_j=\phi(z_j),\quad \operatorname{ReLU}(z)=\max(0,z).

A dense layer with d inputs and h outputs has dh+h parameters when it includes a bias. A network's parameter count adds the counts of its distinct trainable tensors.

Worked example

For x=[1,2], w=[0.5,−1] and b=0.5, z=−1 and ReLU(z)=0. For a 3→4→2 dense network, the first layer has 16 parameters and the second has 4×2+2=10. The total is 26, excluding any other learned normalization or embeddings.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w,b = [1,2],[.5,-1],.5
z = sum(a*v for a,v in zip(w,x))+b
assert z == -1 and max(0,z) == 0
assert (3*4+4)+(4*2+2) == 26

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Stacking affine maps without nonlinearities does not create a nonlinear predictor. A bias is not multiplied by an input feature. Architecture capacity is not proof of predictive usefulness.