D4.2 · LESSON 2 OF 6

Autograd

Translate an inspected NumPy model into tensors, modules, automatic differentiation and a clear training loop.

THE IDEA

Tensors that remember how they were made

Mark a tensor with requires_grad=True and PyTorch records every operation on it, building the computation graph from the previous chapter as you go. Calling loss.backward() on a scalar result walks that graph backwards and stores each leaf tensor's gradient in its .grad.

Only tensors you created with requires_grad=True (the leaves) receive a .grad. Inputs and intermediate results do not keep one.
backward() adds to .grad. Two backward calls leave the sum of both.
After backward() the saved values of that graph are freed. To call it again you need a fresh forward pass (or retain_graph=True).
Anything that leaves PyTorch breaks the history: .item(), .numpy(), converting to a Python number, or .detach(). torch.no_grad() stops recording on purpose.

w = torch.tensor(3.0, requires_grad=True) x = torch.tensor(2.0, requires_grad=False) b = torch.tensor(1.0, requires_grad=True) loss = (w * x + b - 5) ** 2 # (3·2 + 1 − 5)² = 4

requires_grad: wxb

(no actions yet)

tensor	.grad	meaning
w	None	∂loss/∂w = 2(w·x+b−5)·x = 2·2·2 = 8 per backward
x	None	∂loss/∂x = 2(w·x+b−5)·w = 12 per backward
b	None	∂loss/∂b = 2(w·x+b−5) = 4 per backward

A tiny loss (w·x + b − 5)². Toggle which tensors require grad and play the actions in any order. Errors are the ones PyTorch raises.

Predict first. You convert an intermediate result to a plain Python number with .item() and keep computing with it. Is the autograd connection preserved?

real PyTorch (run it locally)

import torch
w = torch.tensor(3.0, requires_grad=True)
x = torch.tensor(2.0)
b = torch.tensor(1.0, requires_grad=True)
loss = (w * x + b - 5) ** 2
loss.backward()
print(w.grad, b.grad, x.grad)   # tensor(8.) tensor(4.) None

Chapter context

PyTorch provides tensor operations and differentiation infrastructure. The framework removes bookkeeping, but it does not remove the need to understand dimensions, parameter ownership, device placement and evaluation behavior.

TRY IT YOURSELF

Part 1. Do it hands-on

PyTorch itself cannot run in a browser here, so the Python parts use a small PyTorch-style library that is already loaded. It has the same names and call shapes (torch.tensor, nn.Linear, loss.backward(), optimizer.step(), DataLoader) and the same behaviour for everything these lessons teach: shapes, accumulating .grad, graphs freed after backward(), parameter registration. It is not PyTorch: initial values and random numbers differ, and "cuda" is only a label. Every code block marked real PyTorch is for you to run on your own machine.

The lab below behaves exactly like the library you will code against. Make it produce each of the situations the tasks describe.

w = torch.tensor(3.0, requires_grad=True) x = torch.tensor(2.0, requires_grad=False) b = torch.tensor(1.0, requires_grad=True) loss = (w * x + b - 5) ** 2 # (3·2 + 1 − 5)² = 4

requires_grad: wxb

(no actions yet)

tensor	.grad	meaning
w	None	∂loss/∂w = 2(w·x+b−5)·x = 2·2·2 = 8 per backward
x	None	∂loss/∂x = 2(w·x+b−5)·w = 12 per backward
b	None	∂loss/∂b = 2(w·x+b−5) = 4 per backward

To do: Run one forward and one backward so that w and b get gradients but x stays None (x does not require grad)
To do: Make w.grad accumulate to 16: two complete forward-and-backward rounds with no zero_grad in between
To do: Now reset between rounds: two backward calls, a zero_grad between them, and w.grad ends at 8 (not 16)
To do: Trigger the 'second time' error on purpose: call backward twice on the same forward pass
To do: Do two backward calls on ONE forward pass without an error, by asking to keep the graph (retain_graph=True) the first time. w.grad ends at 16

Part 2. Do it in code

grad_of(f, x0): x0 is a list of numbers. Make x = torch.tensor(x0, requires_grad=True), compute loss = f(x) (a scalar tensor), call loss.backward() and return x.grad.data as a numpy array.
accumulated_grad(f, x0, times): repeat 'new forward, backward' `times` times on the SAME x without ever clearing x.grad. Return x.grad.data.
descend(f, x0, lr, steps): plain gradient descent by hand. Each step: loss = f(x); loss.backward(); x.data = x.data - lr * x.grad.data; then reset x.grad = None. Return x.data.

import torch

def grad_of(f, x0):
    pass

def accumulated_grad(f, x0, times):
    pass

def descend(f, x0, lr, steps):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does converting an intermediate to an ordinary Python number preserve its autograd connection?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a batch X of shape (B,d), a dense output with h features has shape (B,h). torch.nn.Linear stores weight with shape (h,d) and computes X times weight-transpose plus bias. Parameter count is d*h+h. PyTorch execution belongs in a linked notebook or supported runtime; the chapter does not assume browser PyTorch.

Worked example

A batch has shape (2,4). A Linear(4,3) layer stores a (3,4) weight and a length-three bias. Its output is (2,3). The parameter count is 4×3+3=15. This differs from the batch's element count: model parameters are shared across both examples.

Reference code

The following example uses the Python standard library and does not call a remote service.

batch, inputs, outputs = 2,4,3
weight_shape = (outputs,inputs)
output_shape = (batch,outputs)
assert weight_shape == (3,4)
assert output_shape == (2,3)
assert inputs*outputs+outputs == 15

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

eval mode and no_grad are independent. Model parameters do not multiply with batch size. A tensor detached for logging must not replace the tensor used to compute the training loss.