D3.2 · LESSON 2 OF 6

Initialization

Diagnose an unstable training run using losses, gradient magnitudes, initialization, normalization and regularization.

THE IDEA

The scale of the starting weights decides everything

Before training, the weights are random numbers. Their size matters a great deal in a deep network, because each layer multiplies the signal by roughly (weight size × square root of the number of inputs). If that factor is below 1 the signal shrinks at every layer and vanishes; above 1 it grows and explodes. The same happens to the gradient on the way back.

Two standard recipes pick the scale so the factor stays near 1: Xavier/Glorot (std = sqrt(2 / (fan_in + fan_out)), designed for tanh and sigmoid) and He (std = sqrt(2 / fan_in), designed for ReLU, which zeroes half its inputs and so needs larger weights).

init activation

xavier: std sqrt(2 / (fan_in + fan_out)), made for tanh and sigmoid. Width 16, 8 hidden layers.

last hidden layer / first hidden layer, activation size: 0.531 first-layer gradient RMS at initialisation: 8.0e-2

Width 16, 8 hidden layers. In a ReLU network, He keeps the last layer's signal at 0.67 of the first layer's, while Xavier lets it shrink to 0.059. Try large weights too.

Predict first. You build a 20-layer ReLU network and initialise every weight from a normal distribution with standard deviation 1. What do you expect at the output?

Never start all units the same

There is a second reason to initialise randomly. Two hidden units that start with identical weights compute the same value, receive the same gradient, and are updated identically, so they stay identical forever. A layer of 8 identical units is really 1 unit. Randomness breaks the tie.

start from:

Pick a start to train a 2-8-1 tanh network with Adam for 200 steps.

Train the same 2-8-8-1 network from different starts. A symmetric start leaves the units as exact copies.

Chapter context

A differentiable model can still be difficult to train. Learning rate, parameter scale, data scale and optimizer state interact. Inspect the run before changing many settings at once.

TRY IT YOURSELF

Part 1. Do it hands-on

Use the same lab to find an initialisation that keeps a deep ReLU network healthy, then break and fix the symmetry of a small one.

init activation

hidden layers2

xavier: std sqrt(2 / (fan_in + fan_out)), made for tanh and sigmoid. Width 16, 2 hidden layers.

last hidden layer / first hidden layer, activation size: 0.823 first-layer gradient RMS at initialisation: 4.8e-2

start from:

Pick a start to train a 2-8-1 tanh network with Adam for 200 steps.

To do: Build a ReLU network 8 or more layers deep whose last hidden layer has an activation size between 0.25x and 4x of the first hidden layer's (the signal survives the depth)
To do: Find a setting 8 or more layers deep where the signal shrinks: the last hidden layer's activations are under 10% of the first layer's
To do: Find a setting 8 or more layers deep where the signal explodes: the last hidden layer's activations are at least 100x the first layer's
To do: Train a small network from an initialisation that makes all its hidden units identical. Confirm they never differ (spread exactly 0) and the accuracy stays under 85%
To do: Now train from a random start that breaks the symmetry: the hidden units end up different (spread above 0.5) and the accuracy reaches at least 95%

Part 2. Do it in code

he_std(fan_in): sqrt(2 / fan_in). xavier_std(fan_in, fan_out): sqrt(2 / (fan_in + fan_out)).
init_weights(fan_in, fan_out, kind, seed): a (fan_in, fan_out) array. Use rng = np.random.default_rng(seed); for 'he' and 'xavier' return rng.standard_normal((fan_in, fan_out)) * the matching std; for 'zeros' return zeros. Any other kind, or a non-positive size, raises ValueError.
columns_identical(W): True when all the columns of W are equal to the first one (the hidden units of this layer are exact copies of each other).

import numpy as np

def he_std(fan_in):
    pass

def xavier_std(fan_in, fan_out):
    pass

def init_weights(fan_in, fan_out, kind, seed):
    pass

def columns_identical(W):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why avoid identical zero initialization for all hidden weights in an ordinary MLP?

AIt makes the loss start at exactly zero.BIt makes the first epoch slower but changes nothing else.CIt can preserve symmetry between hidden units: identical units get identical gradients, so they never become different.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Norm clipping uses

g'=g\min\left(1,\frac{c}{\|g\|_2}\right).

For inverted dropout with keep probability q, $\tilde h=m h/q$ where $m\sim\operatorname{Bernoulli}(q)$ , so $E[\tilde h]=h$ . This expectation is over masks, not a claim that each individual masked output is unchanged.

Worked example

The gradient [3,4] has norm 5. Clipping its norm to 2 yields [1.2,1.6], preserving direction. For a scalar activation 4 and keep probability 0.5, dropout gives either 8 or 0 with equal probability, averaging to 4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
g = [3,4]
scale = min(1, 2/sqrt(sum(v*v for v in g)))
clipped = [v*scale for v in g]
assert all(isclose(a,b) for a,b in zip(clipped,[1.2,1.6]))
assert isclose(sqrt(sum(v*v for v in clipped)),2)
assert .5*8+.5*0 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clipping is not the same as normalizing every gradient to the limit. Lower training loss can accompany worse validation. A numerical fix should not conceal invalid data or labels.