D3.1 · LESSON 1 OF 6

Loss and update contracts

Diagnose an unstable training run using losses, gradient magnitudes, initialization, normalization and regularization.

THE IDEA

A training step is four promises

Every training step does the same four things in order: forward (compute predictions), loss (score them against the labels), backward (compute how the loss changes with every weight), and step (move the weights a little against the gradient). Each stage makes a promise to the next one, and most training bugs are a broken promise.

the contract

zero_grad()                    # old gradients must not leak into this step
logits = model(x)              # the loss expects raw scores (logits)...
loss = loss_fn(logits, y)      # ...and applies sigmoid or softmax itself
loss.backward()                # gradients point UPHILL (the direction the loss increases)
optimizer.step()               # so the update subtracts them

Break any line and the loop still runs without an error. It just trains badly, so you need to recognise the symptoms. Here is the same small network on the same data with one thing broken at a time:

training lossvalidation loss

SGD, learning rate 0.5, 150 steps start loss 0.636 → end loss 0.197 accuracy 95% dashboard: healthy: the loss falls and settles

Click through the failures. Healthy training ends near a loss of 0.20. Notice how different the curves look.

The one that looks most innocent is sigmoid applied twice: the model already ends in a sigmoid, and the loss applies another. Training does fall, but only to 0.59 instead of 0.20, because the double squash flattens the gradient.

Predict first. Your classifier ends with a sigmoid layer and you train it with a loss function that already applies a sigmoid to its input (a "with logits" loss). What happens?

Measure both training and validation

The training loss tells you whether the optimiser is working. The validation loss tells you whether the model generalises. Track both from the start: a falling training loss with a rising validation loss is a different problem from a loss that will not fall at all.

Chapter context

A differentiable model can still be difficult to train. Learning rate, parameter scale, data scale and optimizer state interact. Inspect the run before changing many settings at once.

TRY IT YOURSELF

Part 1. Do it hands-on

Four training runs, each with a different bug planted. Read each loss curve and name the bug.

Run A

loss (capped at 20)

start 0.64 → end 1.26 accuracy 78%

Run B

loss (capped at 20)

start 0.64 → end 306.27 accuracy 29%

Run C

loss (capped at 20)

start 0.64 → end 0.64 accuracy 60%

Run D

loss (capped at 20)

start 0.70 → end 0.59 accuracy 83%

To do: Run A: name the bug from its loss curve
To do: Run B: name the bug from its loss curve
To do: Run C: name the bug from its loss curve
To do: Run D: name the bug from its loss curve

Part 2. Do it in code

sigmoid(z): the logistic function, safe for z = ±1000 (no overflow warning). Work with numpy floats.
bce_with_logits(z, y): binary cross-entropy computed straight from the logit z: max(z, 0) - z*y + log(1 + exp(-|z|)). It must give a finite, correct answer for z = ±1000.
bce_from_prob(p, y): binary cross-entropy from a probability. Clamp p to [1e-12, 1 - 1e-12] first so that p = 0 or 1 cannot produce infinity.
sgd_step(params, grads, lr): return a NEW list of arrays param - lr * grad. Never modify the input arrays. Raise ValueError if the two lists have different lengths.

import numpy as np

def sigmoid(z):
    pass

def bce_with_logits(z, y):
    pass

def bce_from_prob(p, y):
    pass

def sgd_step(params, grads, lr):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should logits be converted to probabilities twice before a logits-based cross-entropy function?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Norm clipping uses

g'=g\min\left(1,\frac{c}{\|g\|_2}\right).

For inverted dropout with keep probability q, $\tilde h=m h/q$ where $m\sim\operatorname{Bernoulli}(q)$ , so $E[\tilde h]=h$ . This expectation is over masks, not a claim that each individual masked output is unchanged.

Worked example

The gradient [3,4] has norm 5. Clipping its norm to 2 yields [1.2,1.6], preserving direction. For a scalar activation 4 and keep probability 0.5, dropout gives either 8 or 0 with equal probability, averaging to 4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
g = [3,4]
scale = min(1, 2/sqrt(sum(v*v for v in g)))
clipped = [v*scale for v in g]
assert all(isclose(a,b) for a,b in zip(clipped,[1.2,1.6]))
assert isclose(sqrt(sum(v*v for v in clipped)),2)
assert .5*8+.5*0 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Clipping is not the same as normalizing every gradient to the limit. Lower training loss can accompany worse validation. A numerical fix should not conceal invalid data or labels.