D3.4 · LESSON 4 OF 6

Normalization

Diagnose an unstable training run using losses, gradient magnitudes, initialization, normalization and regularization.

THE IDEA

Rescale activations inside the network

Training is easier when the numbers entering a layer have a steady scale. Normalisation layers enforce that: subtract a mean, divide by a standard deviation, so the layer sees values with mean 0 and variance 1 (then usually learn a scale and shift on top). The difference between the kinds is only which numbers the mean and variance are computed over.

Batch normalisation computes one mean and variance per feature, over all examples in the batch.
Layer normalisation computes one mean and variance per example, over that example's features.

feature

input (4 examples × 3 features)

1	10	100
3	20	200
5	30	300
7	40	400

after batch norm (no epsilon, no scale or shift)

-1.342	-1.342	-1.342
-0.447	-0.447	-0.447
0.447	0.447	0.447
1.342	1.342	1.342

feature 1: values 1, 3, 5, 7 mean = 4 variance = 5 → each value becomes (x − mean) / sqrt(variance)

The same 4×3 table, normalised along different axes. The highlighted group is the set of numbers that shared one mean and variance.

Why they behave differently at inference

Batch norm depends on the other examples in the batch. That is fine in training, but at inference you may predict one example at a time, so it uses running averages of the mean and variance collected during training (this is the "eval mode" of a batch norm layer). Layer norm needs no such state, because it looks only inside the example. With a batch of one example, batch norm has no spread to measure; layer norm still works.

Predict first. Are batch normalisation and layer normalisation interchangeable operations?

Chapter context

A differentiable model can still be difficult to train. Learning rate, parameter scale, data scale and optimizer state interact. Inspect the run before changing many settings at once.

TRY IT YOURSELF

Part 1. Do it hands-on

Do the normalisation arithmetic by hand on a small table, then choose the right axis.

batch (4 examples × 3 features): [ 1, 10, 100] [ 3, 20, 200] [ 5, 30, 300] [ 7, 40, 400]

mean of feature 1 variance of feature 1

batch-normalised value of 3 (example 2, feature 1)

layer-normalised value of 100 (example 1, feature 3)

Batch norm averages over:

One example only: the sensible choice is

To do: Batch normalisation works per feature, over the batch. For the first feature (1, 3, 5, 7) type its mean and its variance (divide by 4)
To do: Normalise the second example's first feature (the value 3) with those statistics (ignore the tiny epsilon)
To do: Layer normalisation works per example, over its features. Normalise the third feature (the value 100) of the first example (1, 10, 100)
To do: Which axis does batch normalisation average over?
To do: A batch contains just ONE example. Which of the two still gives a meaningful normalisation of it?

Part 2. Do it in code

batch_norm(X, eps=1e-5): X has shape (batch, features). Normalise every FEATURE over the batch: (X - mean) / sqrt(var + eps), with the mean and the (population) variance taken along axis 0.
layer_norm(X, eps=1e-5): normalise every EXAMPLE over its features, along axis 1.
batch_norm_infer(X, mean, var, eps=1e-5): inference mode. Use the stored running mean and variance instead of statistics from this batch.

import numpy as np

def batch_norm(X, eps=1e-5):
    pass

def layer_norm(X, eps=1e-5):
    pass

def batch_norm_infer(X, mean, var, eps=1e-5):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Are batch normalization and layer normalization interchangeable operations?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Norm clipping uses

g'=g\min\left(1,\frac{c}{\|g\|_2}\right).

For inverted dropout with keep probability q, $\tilde h=m h/q$ where $m\sim\operatorname{Bernoulli}(q)$ , so $E[\tilde h]=h$ . This expectation is over masks, not a claim that each individual masked output is unchanged.

Worked example

The gradient [3,4] has norm 5. Clipping its norm to 2 yields [1.2,1.6], preserving direction. For a scalar activation 4 and keep probability 0.5, dropout gives either 8 or 0 with equal probability, averaging to 4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
g = [3,4]
scale = min(1, 2/sqrt(sum(v*v for v in g)))
clipped = [v*scale for v in g]
assert all(isclose(a,b) for a,b in zip(clipped,[1.2,1.6]))
assert isclose(sqrt(sum(v*v for v in clipped)),2)
assert .5*8+.5*0 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clipping is not the same as normalizing every gradient to the limit. Lower training loss can accompany worse validation. A numerical fix should not conceal invalid data or labels.