Concurrent sequences
Calculate KV-cache memory and explain how layers, KV heads, context length and concurrent sequences affect it.
THE IDEA
Many conversations, many caches
A server handles many sequences at once. Each has its own text, so each has its own keys and values. Four independent 4096-token sequences need four times the memory of one: 2 GiB for the reference model. The only way to save is when sequences really do share an identical, compatible prefix (for example the same system prompt), which can be stored once.
Batching processes many sequences in one step, but it does not merge their conversations: unless a prefix is explicitly and safely shared, each sequence needs its own state, so memory scales with the number of sequences.
Predict first. Does doubling concurrent unshared sequences double this simple cache estimate?
Chapter context
Causal attention can reuse keys and values for earlier tokens rather than recomputing them at every generated step. This saves work but creates state that grows with the number of retained token positions.
TRY IT YOURSELF
Part 1. Do it hands-on
Total up concurrent caches with and without a shared prefix, then write the budget helpers.
Batching processes many sequences in one step, but it does not merge their conversations: unless a prefix is explicitly and safely shared, each sequence needs its own state, so memory scales with the number of sequences.
- To do: Four independent sequences of 4096 tokens each (512 MiB each). Total in MiB?
- To do: How many 4096-token sequences fit in a 16 GiB (16,384 MiB) KV budget?
- To do: Four sequences share an identical 1024-token prefix that is stored ONCE; each then has 3072 tokens of its own. Total MiB?
Part 2. Do it in code
- total_tokens(lengths): the sum of independent sequence lengths (unshared state).
- total_tokens_shared(prefix_len, suffix_lens): prefix_len (stored once) plus the sum of the suffix lengths.
- total_bytes(tokens, per_token): tokens · per_token.
- max_sequences(budget_bytes, seq_tokens, per_token): how many unshared sequences of seq_tokens fit in the budget.
- sharing_savings(prefix_len, suffix_lens): tokens saved by storing the prefix once instead of in every sequence.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does doubling concurrent unshared sequences double this simple cache estimate?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
For a uniform full-attention decoder:
L is layers, H_KV KV heads, d_h head dimension, T retained tokens per sequence, B equal-length unshared sequences and b bytes per value. The factor 2 counts keys and values. For unequal lengths, replace TB by the sum of retained lengths.
Worked example
For L=32, H_KV=8, d_h=128 and b=2, each retained token needs 2×32×8×128×2=131,072 bytes, or 128 KiB. At T=4096, one sequence needs 512 MiB. Four equally long independent sequences need 2 GiB, before allocator overhead and other memory.
Reference code
The following example uses the Python standard library and does not call a remote service.
layers,heads,dim,bytes_per_value=32,8,128,2
per_token=2*layers*heads*dim*bytes_per_value
assert per_token == 131072
assert per_token/1024 == 128
assert per_token*4096/1024**2 == 512
assert per_token*4096*4/1024**3 == 2
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Use KV heads rather than query heads. Distinguish decimal GB from binary GiB. Hybrid, sliding-window and quantized-cache architectures require their own detailed accounting.