D5.6 · LESSON 6 OF 6

Residual connections

Calculate a convolutional output and explain filters, stride, padding, pooling and residual connections.

THE IDEA

Shortcuts that carry the input around the layers

A residual block computes y = x + F(x): it adds the layers' output F(x) to the unchanged input x. If a block has nothing useful to add it can learn F ≈ 0 and simply pass x on. For gradients, the addition gives each block a derivative of 1 + (slope of F) instead of just the slope, so the gradient through a very deep stack does not collapse to zero.

x = [1,-2,3] F(x) = [0.5,-1,0.5] (what the layers learned) x + F(x) = [1.5,-3,3.5] (the shortcut carries x through unchanged)

slope w of each layer0.4

stack of 20 layers	gradient reaching the first layer
plain: w^20	1.10e-8
residual: (1 + w)^20	8.37e+2

Twenty layers with slope w. The plain stack's gradient is w^20; the residual stack's is (1 + w)^20.

Predict first. Can a shortcut add two tensors of incompatible shapes with no defined mapping between them?

NumPy

F = np.maximum(x @ W1, 0) @ W2      # the learned part
y = F + x                           # the shortcut (or x @ proj when shapes change)

Chapter context

A convolutional layer reuses a local filter across positions. Weight sharing encodes a structural assumption about repeated local patterns while greatly reducing parameters compared with a fully connected map over all pixels.

TRY IT YOURSELF

Part 1. Do it hands-on

Add a shortcut by hand, decide which blocks need a projection and find a slope where only the residual stack survives. Then implement a residual block.

x = [1,-2,3] F(x) = [0.5,-1,0.5] (what the layers learned) x + F(x) = ? (the shortcut carries x through unchanged)

slope w of each layer0.4

stack of 20 layers	gradient reaching the first layer
plain: w^20	1.10e-8
residual: (1 + w)^20	8.37e+2

y = x + F(x):

Which shortcuts need a projection?

64 channels in, 64 out, same spatial size 64 channels in, 128 out 64 channels in and out, but the spatial size is halved (stride 2)

To do: A block computes y = x + F(x) with x = [1, −2, 3] and F(x) = [0.5, −1, 0.5]. Type y
To do: Tick every block whose shortcut needs a projection (for example a 1×1 convolution) before it can be added
To do: 20 layers deep, each with slope w. Choose w so a plain stack's gradient has vanished (below 1e-6) while the residual stack's (1 + w per layer) stays between 0.5 and 100

Part 2. Do it in code

residual_block(x, W1, W2, proj=None): x has shape (n, d). Compute F = relu(x @ W1) @ W2. The shortcut is x itself, or x @ proj when proj is given. Return F + shortcut. If the shapes of F and the shortcut differ raise ValueError('shape mismatch: ...').
chain_gradient(w, layers, residual): the gradient through `layers` identical layers: w ** layers for a plain stack and (1 + w) ** layers when residual is True. layers must be at least 0 (otherwise ValueError).

import numpy as np

def residual_block(x, W1, W2, proj=None):
    pass

def chain_gradient(w, layers, residual):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a shortcut add tensors of incompatible shapes without a defined mapping?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For one spatial dimension, output size is

n_{out}=\left\lfloor\frac{n+2p-d(k-1)-1}{s}\right\rfloor+1.

Here n is input size, k kernel size, p padding, d dilation and s stride. Cross-correlation is $y_t=\sum_iw_i x_{t+i}$ . A residual block has form $h_{out}=h+F(h)$ when shapes agree.

Worked example

For input [1,2,3,4] and kernel [1,−1], stride-one valid cross-correlation returns [−1,−1,−1]. Separately, n=7,k=3,p=1,s=2,d=1 gives floor((7+2−2−1)/2)+1=4 output positions.

Reference code

The following example uses the Python standard library and does not call a remote service.

x,w = [1,2,3,4],[1,-1]
y = [sum(x[t+i]*w[i] for i in range(len(w))) for t in range(len(x)-len(w)+1)]
assert y == [-1,-1,-1]
n,k,p,s,d = 7,3,1,2,1
assert (n+2*p-d*(k-1)-1)//s+1 == 4

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Padding does not reveal missing real pixels. Kernel flipping conventions change calculations. A residual shortcut needs a defined shape-compatible path.