L9.3 · LESSON 3 OF 6

Long context

Explain expert routing, long-context costs, state-space models and multimodal token streams under explicit assumptions.

THE IDEA

What a long context costs

Naive attention compares every query with every key, so the score matrix has n × n entries per head: 16,777,216 for 4096 tokens. Doubling the length quadruples it. The KV cache needed while generating grows only linearly with n, but for long contexts it still dominates memory.

context length n4,096

quantity	formula	value
score entries per head (full)	n²	16,777,216
score entries per head (causal)	n(n+1)/2	8,390,656
KV cache, 2 layers × 2 heads × 64 (numbers)	2·2·2·64·n	2,097,152

Compute for the score matrix grows with the square of the length; the cache grows linearly. Doubling n multiplies the first by 4 and the second by 2.

A sliding window limits each token to its w most recent neighbours (8 tokens, window w):

window w4

pairs computed = 26 (a full causal matrix would have 36)

A model that advertises a long context window can still retrieve badly from some positions. The usable context depends on position encoding, training data and memory techniques, so test retrieval at the positions you care about.

Slide the context length. Below, a sliding window cuts the pair count.

Predict first. Does advertising a larger context guarantee reliable retrieval from every position?

Chapter context

New architectures alter where computation and memory are spent. Compare them by tracing one request through their actual operations rather than assuming every improvement reduces every cost.

TRY IT YOURSELF

Part 1. Do it hands-on

Count attention entries and windowed pairs, then write the cost formulas.

context length n4,096

quantity	formula	value
score entries per head (full)	n²	16,777,216
score entries per head (causal)	n(n+1)/2	8,390,656
KV cache, 2 layers × 2 heads × 64 (numbers)	2·2·2·64·n	2,097,152

Compute for the score matrix grows with the square of the length; the cache grows linearly. Doubling n multiplies the first by 4 and the second by 2.

A sliding window limits each token to its w most recent neighbours (8 tokens, window w):

window w4

pairs computed = 26 (a full causal matrix would have 36)

entries for 4096 tokens times more at 8192 pairs, 8 tokens, window 4

To do: A naive attention score matrix for 4096 tokens has how many entries per head?
To do: Doubling the length to 8192: how many times more entries?
To do: A causal model where each token sees only the 4 most recent tokens (itself included). For 8 tokens, how many (query, key) pairs are computed? (A full causal matrix would have 36.)

Part 2. Do it in code

attention_entries(n, heads=1, causal=False): heads · n² for full attention; with causal=True count only pairs where the key is not after the query: heads · n(n+1)/2.
window_entries(n, w): causal attention where each token sees at most the w most recent tokens (itself included).
growth_factor(n1, n2): how many times more full-attention entries n2 needs than n1.
kv_cache_numbers(layers, kv_heads, head_dim, n): 2 · layers · kv_heads · head_dim · n (grows linearly in n).

def attention_entries(n, heads=1, causal=False):
    pass

def window_entries(n, w):
    pass

def growth_factor(n1, n2):
    pass

def kv_cache_numbers(layers, kv_heads, head_dim, n):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does advertising a larger context guarantee reliable retrieval from every position?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a toy MoE with shared weights S, E equal experts of P weights and k experts active per token:

N_{total}=S+EP,\quad N_{active}\approx S+kP.

This is a parameter-accounting model, not a full FLOP or communication model. Naively materialized attention scores use $T^2$ entries per head.

Worked example

With 2 billion shared parameters and eight experts of 1 billion each, total parameters are 10 billion. If two experts are selected per token, approximately 4 billion parameters participate along that token's path. The other experts still require storage or loading. A 4096×4096 naive score matrix contains 16,777,216 entries per head.

Reference code

The following example uses the Python standard library and does not call a remote service.

shared, experts, per_expert, active=2,8,1,2
assert shared+experts*per_expert == 10
assert shared+active*per_expert == 4
assert 4096**2 == 16777216

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Active parameters and resident parameters differ. Linear sequence complexity does not make every model property linear. Context-window capacity and demonstrated context use are separate claims.