D3.5 · LESSON 5 OF 6

Dropout and schedules

Diagnose an unstable training run using losses, gradient magnitudes, initialization, normalization and regularization.

THE IDEA

Dropout: train a different sub-network each step

Dropout randomly sets a fraction p of the activations to zero during training, so the network cannot rely on any one unit. The survivors are scaled up by 1 / (1 − p) ("inverted" dropout) so the average activation is unchanged. At evaluation dropout is switched off entirely: no zeroing, no scaling, because the average already matches.

drop probability p0.5

unit	1	2	3	4	5	6	7	8	9	10	mean
activation	4	2	6	1	5	3	7	2	4	6	4.00
kept?	no	yes	yes	yes	no	yes	yes	yes	no	no
output (train)	0.00	4.00	12.00	2.00	0.00	6.00	14.00	4.00	0.00	0.00	4.20
output (eval)	4	2	6	1	5	3	7	2	4	6	4.00

Kept values are divided by (1 − p) = 0.5, i.e. multiplied by 2.00. One mask is noisy, but over many masks the average output equals the evaluation output, which is why evaluation can simply switch dropout off.

One mask over ten activations. Draw new masks and compare the mean of the training output with the evaluation output.

Predict first. With keep probability 0.5, what is a kept activation multiplied by in inverted dropout?

dropout p	train loss (dropout off)	validation loss	train accuracy
0	0.000	0.037	100%
0.2	0.007	0.243	100%
0.5	0.017	0.061	99%
0.8	0.383	0.452	82%

A real regularisation effect on a wider network: more dropout means less fitting of the training set.

Learning-rate schedules

The best learning rate changes during training. A warm-up ramps it up from near zero in the first steps, when the gradients and (for Adam) the running statistics are least reliable. Then a decay lowers it so the weights settle into a minimum instead of bouncing around it. Common shapes are a step decay (divide by a fixed factor every N steps) and a cosine curve.

warm-up steps0

learning rate (constant)

peak 0.1, 200 steps: step 10 → 0.1000 step 100 → 0.1000 step 120 → 0.10000 step 199 → 0.100000

Shape the schedule and read the learning rate at a few steps.

Chapter context

A differentiable model can still be difficult to train. Learning rate, parameter scale, data scale and optimizer state interact. Inspect the run before changing many settings at once.

TRY IT YOURSELF

Part 1. Do it hands-on

First the dropout arithmetic, then build two learning-rate schedules to exact specifications.

factor applied to a kept activation (keep probability 0.5)

a kept activation of 4 becomes

warm-up steps0

learning rate (constant)

peak 0.1, 200 steps: step 10 → 0.1000 step 100 → 0.1000 step 120 → 0.10000 step 199 → 0.100000

To do: Inverted dropout with keep probability 0.5: by what factor is every KEPT activation multiplied?
To do: A kept activation has the value 4. What does it become? (the other half become 0, so the average stays 4)
To do: Peak learning rate 0.1 over 200 steps. Build a schedule that warms up for 20 steps and then decays along a cosine to almost nothing: lr at step 10 is 0.055 and lr at step 199 is below 0.001
To do: Now a step schedule with no warm-up: the rate falls to a tenth every 50 steps, so at step 120 it equals 0.001

Part 2. Do it in code

inverted_dropout(x, p, seed, train=True): p is the DROP probability in [0, 1] (otherwise ValueError). If train is False or p == 0 return x unchanged. Otherwise mask = rng.random(x.shape) >= p with rng = np.random.default_rng(seed), and return x * mask / (1 - p). For p == 1 return zeros (no division by zero).
lr_schedule(step, total, peak, warmup=0, kind='cosine', every=50, gamma=0.1): linear warm-up first: if step < warmup return peak * (step + 1) / warmup. Then with u = step - warmup and span = max(1, total - warmup): 'cosine' gives 0.5 * peak * (1 + cos(pi * min(u, span) / span)); 'step' gives peak * gamma ** (u // every); any other kind raises ValueError.

import numpy as np

def inverted_dropout(x, p, seed, train=True):
    pass

def lr_schedule(step, total, peak, warmup=0, kind="cosine", every=50, gamma=0.1):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

With keep probability .5, what scaling is applied to a kept activation in inverted dropout?

AMultiply by two (1 / 0.5), so the average activation stays the same as without dropout.BMultiply by 0.5, so the kept values shrink to match the dropped ones.CMultiply by 1, because dropout only sets some values to zero.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Norm clipping uses

g'=g\min\left(1,\frac{c}{\|g\|_2}\right).

For inverted dropout with keep probability q, $\tilde h=m h/q$ where $m\sim\operatorname{Bernoulli}(q)$ , so $E[\tilde h]=h$ . This expectation is over masks, not a claim that each individual masked output is unchanged.

Worked example

The gradient [3,4] has norm 5. Clipping its norm to 2 yields [1.2,1.6], preserving direction. For a scalar activation 4 and keep probability 0.5, dropout gives either 8 or 0 with equal probability, averaging to 4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
g = [3,4]
scale = min(1, 2/sqrt(sum(v*v for v in g)))
clipped = [v*scale for v in g]
assert all(isclose(a,b) for a,b in zip(clipped,[1.2,1.6]))
assert isclose(sqrt(sum(v*v for v in clipped)),2)
assert .5*8+.5*0 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clipping is not the same as normalizing every gradient to the limit. Lower training loss can accompany worse validation. A numerical fix should not conceal invalid data or labels.