L5.1 · LESSON 1 OF 6

Residual streams

Trace a transformer block, distinguish normalization variants and count a simplified block's parameters.

THE IDEA

One stream, many small edits

A transformer block does not transform a token's vector from scratch. Think of each token as carrying a residual stream, a vector that travels through the whole model. Every sublayer (attention, then the MLP) reads the stream, computes an update, and adds it back: x ← x + update.

Two consequences: whatever a sublayer leaves alone passes straight through, and the update must have exactly the stream's shape because addition works coordinate by coordinate.

attention adds its update MLP adds its update make the MLP update 4 numbers wide

step	vector
stream x	[1, 2, 3]
+ attention update	[0.5, -1, 0]
= stream after attention	[1.5, 1, 3]
+ MLP update	[0, 1, 1]
= stream after MLP	[1.5, 2, 4]

Each sublayer adds to the stream instead of replacing it, so information it does not touch passes through unchanged. Addition needs the same shape, which is why every sublayer outputs the model width.

Toggle each sublayer. A sublayer with nothing to add changes nothing.

Predict first. Why must an added update match the residual stream shape?

NumPy

x = x + attention(x)     # sublayer 1 adds an update
x = x + mlp(x)           # sublayer 2 adds another

Chapter context

A transformer combines attention with per-position feed-forward computation and residual paths. Exact ordering and normalization differ between architectures; learn a stated variant instead of assuming every transformer has an identical block.

TRY IT YOURSELF

Part 1. Do it hands-on

Trace a stream through two sublayers by hand, then write residual addition with a strict shape check.

attention adds its update MLP adds its update make the MLP update 4 numbers wide

step	vector
stream x	[1, 2, 3]
+ attention update	[0.5, -1, 0]
= stream after attention	[1.5, 1, 3]
+ MLP update	[0, 1, 1]
= stream after MLP	[1.5, 2, 4]

after attention:

after the MLP:

with a zero attention update, after the MLP:

To do: The stream is x = [1, 2, 3]. The attention sublayer adds the update [0.5, −1, 0]. Type the stream afterwards
To do: Then the MLP sublayer adds [0, 1, 1]. Type the stream after both sublayers
To do: Suppose the attention update is all zeros (it has nothing to add). What is the stream after both sublayers?

Part 2. Do it in code

add_residual(x, update): return x + update. Raise ValueError if the shapes differ (no broadcasting allowed).
run_sublayers(x, fns): apply each function in order as x = x + fn(x) and return the final stream.
trace(x, fns): the list [x, after first sublayer, after second, ...] (length len(fns) + 1).
total_update(x, fns): the final stream minus the starting x (everything the sublayers contributed).

import numpy as np

def add_residual(x, update):
    pass

def run_sublayers(x, fns):
    pass

def trace(x, fns):
    pass

def total_update(x, fns):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why must an added update match the residual stream shape?

AAttention always outputs a single scalar per token.BBecause addition works coordinate by coordinate, so a 4-number update cannot be added to a 3-number stream; every sublayer projects back to the model width.CThe optimizer refuses updates of different shapes.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

One possible pre-norm block is

h=x+\operatorname{Attention}(\operatorname{Norm}(x)),\quad y=h+\operatorname{MLP}(\operatorname{Norm}(h)).

For standard full-width Q/K/V/O projections and an ungated d→f→d MLP, matrix weights total $4d^2+2df$ , excluding bias and normalization. GQA changes K/V projection dimensions.

For one feature vector x of width d, LayerNorm uses $\mu=\frac1d\sum_i x_i$ and $v=\frac1d\sum_i(x_i-\mu)^2$ , then normalizes $(x_i-\mu)/\sqrt{v+\epsilon}$ before its learned affine transform. RMSNorm instead scales using $\sqrt{\frac1d\sum_i x_i^2+\epsilon}$ without the same mean subtraction. Neither definition normalizes across arbitrary axes unless those axes are part of the specified layer.

Worked example

Let d=4 and f=8. Four attention matrices contain 4×4²=64 weights. The two MLP matrices contain 2×4×8=64. The simplified block has 128 matrix weights. Embedding tables, learned normalization scales, biases and any positional parameters are outside this count.

For x=[3,4] without affine transforms or epsilon, LayerNorm produces [-1,1], while RMSNorm divides by sqrt(12.5). The outputs differ because RMSNorm does not center the vector.

Reference code

The following example uses the Python standard library and does not call a remote service.

width,hidden=4,8
attention=4*width*width
mlp=2*width*hidden
assert attention == 64 and mlp == 64
assert attention+mlp == 128
query_heads,kv_heads=4,2
assert query_heads//kv_heads == 2
from math import sqrt,isclose
x=[3.,4.]
mean=sum(x)/len(x)
var=sum((v-mean)**2 for v in x)/len(x)
assert [(v-mean)/sqrt(var) for v in x] == [-1.,1.]
rms=sqrt(sum(v*v for v in x)/len(x))
assert isclose(sum((v/rms)**2 for v in x)/len(x),1)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Parameter-count formulas depend on the architecture. RMSNorm and LayerNorm do not perform identical transformations. Smaller KV caches do not mean the model has no attention or no queries.