D3.6 · LESSON 6 OF 6

Gradient health

Diagnose an unstable training run using losses, gradient magnitudes, initialization, normalization and regularization.

THE IDEA

Products of small or large numbers

Backpropagation multiplies local derivatives together, one per layer. Multiply many numbers below 1 and the gradient reaching the early layers is tiny (vanishing); multiply many above 1 and it is huge (exploding). Sigmoid is a classic culprit: its slope never exceeds 0.25.

activation init

hidden layers2

training loss

SGD, learning rate 0.3, 100 steps, width 16 first-layer gradient RMS: during training 1.4e-3 at initialisation 4.8e-2 final training loss 0.358 spiking: sudden jumps in the loss

Eight hidden layers, SGD. With sigmoid the first layer's gradient is about 1.2e-6 and training barely moves (loss 0.69). With ReLU and He initialisation it is about 2.5e-3 and the loss reaches 0.061.

Clipping the damage

When one bad batch produces a giant gradient, a single step can wreck the weights. Gradient clipping rescales the whole gradient whenever its norm exceeds a threshold, keeping its direction but limiting its size. For a gradient [3, 4] the norm is 5; clipping to 2 multiplies by 2/5 = 0.4 and gives [1.2, 1.6].

clip threshold on the gradient normoff

training loss

gradient norm before clippingafter clipping

batch inputs multiplied by 30 at steps 60 and 120 (a corrupted batch) dashboard: spiking: sudden jumps in the loss final loss 0.0055

A corrupted batch (inputs ×30) hits at steps 60 and 120. Move the threshold and compare the gradient norm before and after clipping.

Predict first. The gradient is [3, 4] and the norm limit is 2. What scale factor is applied?

Chapter context

A differentiable model can still be difficult to train. Learning rate, parameter scale, data scale and optimizer state interact. Inspect the run before changing many settings at once.

TRY IT YOURSELF

Part 1. Do it hands-on

Diagnose and repair a deep network, then tame a run with gradient spikes.

activation init

hidden layers2

training loss

SGD, learning rate 0.3, 100 steps, width 16 first-layer gradient RMS: during training 1.4e-3 at initialisation 4.8e-2 final training loss 0.358 spiking: sudden jumps in the loss

clip threshold on the gradient normoff

training loss

gradient norm before clippingafter clipping

batch inputs multiplied by 30 at steps 60 and 120 (a corrupted batch) dashboard: spiking: sudden jumps in the loss final loss 0.0055

To do: Find a setting 8 or more layers deep where the first layer's gradient RMS during training is below 1e-5 (the early layers have stopped learning)
To do: Now fix it: 8 or more layers deep, first-layer gradient RMS above 1e-3 AND the final training loss below 0.1
To do: At initialisation, find a deep network (8+ layers) whose first-layer gradient RMS is 100 or more (exploding gradients)
To do: In the spiky run (the inputs are multiplied by 30 at steps 60 and 120) choose a clip threshold that brings the diagnosis to ok

Part 2. Do it in code

global_norm(grads): grads is a list of arrays. Return the square root of the sum of squares of ALL their entries, as one number.
clip_by_global_norm(grads, max_norm): return (clipped, norm). If norm > max_norm every array is multiplied by max_norm / norm, otherwise the arrays are returned unchanged (as new arrays). max_norm <= 0 raises ValueError. A norm of 0 must not divide by zero. Never modify the inputs.
has_bad_values(grads): True if any entry of any array is NaN or infinite.

import numpy as np

def global_norm(grads):
    pass

def clip_by_global_norm(grads, max_norm):
    pass

def has_bad_values(grads):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For gradient [3,4] and norm limit 2, what scale factor applies?

A2/5 = 0.4, because the norm is 5 and the limit is 2.B2.5, because the norm 5 is divided by the limit 2.C0.5, because the limit 2 is divided by 4.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Norm clipping uses

g'=g\min\left(1,\frac{c}{\|g\|_2}\right).

For inverted dropout with keep probability q, $\tilde h=m h/q$ where $m\sim\operatorname{Bernoulli}(q)$ , so $E[\tilde h]=h$ . This expectation is over masks, not a claim that each individual masked output is unchanged.

Worked example

The gradient [3,4] has norm 5. Clipping its norm to 2 yields [1.2,1.6], preserving direction. For a scalar activation 4 and keep probability 0.5, dropout gives either 8 or 0 with equal probability, averaging to 4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
g = [3,4]
scale = min(1, 2/sqrt(sum(v*v for v in g)))
clipped = [v*scale for v in g]
assert all(isclose(a,b) for a,b in zip(clipped,[1.2,1.6]))
assert isclose(sqrt(sum(v*v for v in clipped)),2)
assert .5*8+.5*0 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clipping is not the same as normalizing every gradient to the limit. Lower training loss can accompany worse validation. A numerical fix should not conceal invalid data or labels.