D4.6 · LESSON 6 OF 6

Devices and debugging

Translate an inspected NumPy model into tensors, modules, automatic differentiation and a clear training loop.

THE IDEA

Devices and dtypes must agree

A model's parameters and the tensors it processes must live on the same device. Move a model to a GPU and the input stays on the CPU, and the next call fails. The same goes for element types (a float model cannot take an integer input without a conversion). These errors are not subtle once you know to read them.

model is oninput x is on

model = model.to('cpu') x = x.to('cpu') out = model(x) # fine: everything is on cpu

"cuda:0" here is only a label: the exercises have no GPU. The rule being taught is the real one: the model's parameters and every tensor it touches must be on the same device.

Move the model and the input independently. ("cuda:0" is a label in this simulation.)

real PyTorch (run it locally)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
for xb, yb in loader:
    xb, yb = xb.to(device), yb.to(device)   # move every batch too
    ...

Debug small, then scale up

When something fails, shrink the problem: take one tiny batch, run it through the model layer by layer and print each shape. A bigger GPU only runs the same wrong shapes faster.

Predict first. Will a bigger GPU repair an incorrect matrix dimension?

Chapter context

PyTorch provides tensor operations and differentiation infrastructure. The framework removes bookkeeping, but it does not remove the need to understand dimensions, parameter ownership, device placement and evaluation behavior.

TRY IT YOURSELF

Part 1. Do it hands-on

PyTorch itself cannot run in a browser here, so the Python parts use a small PyTorch-style library that is already loaded. It has the same names and call shapes (torch.tensor, nn.Linear, loss.backward(), optimizer.step(), DataLoader) and the same behaviour for everything these lessons teach: shapes, accumulating .grad, graphs freed after backward(), parameter registration. It is not PyTorch: initial values and random numbers differ, and "cuda" is only a label. Every code block marked real PyTorch is for you to run on your own machine.

Five real error situations. For each, pick the fix that matches what the message says.

Error 1

layer = nn.Linear(5, 3)
out = layer(x)   # x has shape (32, 10)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x10 and 5x3)

Error 2

model = model.to('cuda:0')
out = model(x)   # x was created on the CPU

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Error 3

out = model(x)       # shape (32, 1)
out.backward()

RuntimeError: grad can be implicitly created only for scalar outputs

Error 4

loss = loss_fn(model(x), y)
loss.backward()
loss.backward()

RuntimeError: Trying to backward through the graph a second time ...

Silent bug

w = torch.tensor(1.0, requires_grad=True)
w = w.to('cuda:0')
(w * 2).backward()
print(w.grad)   # None

No error, but w.grad is None

To do: Error 1: choose the fix
To do: Error 2: choose the fix
To do: Error 3: choose the fix
To do: Error 4: choose the fix
To do: Silent bug: choose the fix

Part 2. Do it in code

shape_trace(layers, in_shape): layers is a list like [('linear', 10, 4), ('relu',), ('linear', 4, 3)] and in_shape is the shape of the input batch, for example (32, 10). Return the list of output shapes after every layer. A ('linear', i, o) layer needs the last dimension to equal i and changes it to o; ('relu',) keeps the shape. If the last dimension does not match, raise ValueError whose message names the layer number (counting from 0), for example 'layer 0: expected last dim 5, got 10'.
same_device(*tensors): return the common device string of all the tensors, or raise RuntimeError whose message contains both device names when they differ.
safe_backward(loss): if loss has more than one element, reduce it with loss.mean() first. Then call backward() on the scalar and return the scalar's value as a float. If loss is not part of a graph (it does not require grad) raise ValueError('loss does not require grad').

import torch

def shape_trace(layers, in_shape):
    pass

def same_device(*tensors):
    pass

def safe_backward(loss):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Will a bigger GPU repair an incorrect matrix dimension?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a batch X of shape (B,d), a dense output with h features has shape (B,h). torch.nn.Linear stores weight with shape (h,d) and computes X times weight-transpose plus bias. Parameter count is d*h+h. PyTorch execution belongs in a linked notebook or supported runtime; the chapter does not assume browser PyTorch.

Worked example

A batch has shape (2,4). A Linear(4,3) layer stores a (3,4) weight and a length-three bias. Its output is (2,3). The parameter count is 4×3+3=15. This differs from the batch's element count: model parameters are shared across both examples.

Reference code

The following example uses the Python standard library and does not call a remote service.

batch, inputs, outputs = 2,4,3
weight_shape = (outputs,inputs)
output_shape = (batch,outputs)
assert weight_shape == (3,4)
assert output_shape == (2,3)
assert inputs*outputs+outputs == 15

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

eval mode and no_grad are independent. Model parameters do not multiply with batch size. A tensor detached for logging must not replace the tensor used to compute the training loss.