E2.5 · LESSON 5 OF 6

Precision and overhead

Calculate KV-cache memory and explain how layers, KV heads, context length and concurrent sequences affect it.

THE IDEA

Precision and hidden overhead

Storing the cache in fewer bits cuts the nominal size: 8 bits instead of 16 halves 128 KiB to 64 KiB per token. But quantised values need scales (for instance a 2-byte scale per group of 64 values adds 2/64 byte per value, taking 64 KiB to 66 KiB), allocators hand out memory in blocks (4097 tokens in blocks of 16 take 4112 slots), and there is workspace besides. The model weights are a separate, usually larger, cost.

bits per value8values per scale (group)64sequence tokens4,097allocator block (tokens)16

nominal bytes per value = 8/8 = 1; with a 2-byte scale per 64 values = 1.0313 per token (65,536 values) = 66 KiB (16-bit reference: 128 KiB) 4,097 tokens in blocks of 16 → 4,112 slots (15 wasted) = 265.03 MiB

Lower precision shrinks the nominal size, but scale metadata, rounding up to allocator blocks and workspace all add memory back. Model weights are a separate cost again.

Change the precision, the group size and the block size.

Predict first. Is KV memory the whole GPU memory footprint?

Chapter context

Causal attention can reuse keys and values for earlier tokens rather than recomputing them at every generated step. This saves work but creates state that grows with the number of retained token positions.

TRY IT YOURSELF

Part 1. Do it hands-on

Account for scale metadata and block rounding by hand, then code the effective-size functions.

bits per value8values per scale (group)64sequence tokens4,097allocator block (tokens)16

Lower precision shrinks the nominal size, but scale metadata, rounding up to allocator blocks and workspace all add memory back. Model weights are a separate cost again.

KiB per token at 8 bits KiB with a scale per 64 values slots for 4097 tokens, block 16

To do: Storing the same cache in 8 bits (1 byte) per value instead of 16: KiB per token for the reference model?
To do: Quantisation also stores a 2-byte scale for every group of 64 values. That adds 2/64 bytes per value on top of the 1 byte. KiB per token now? (The model has 65,536 values per token.)
To do: Memory is handed out in blocks of 16 tokens. A sequence of 4097 tokens needs how many token slots?

Part 2. Do it in code

effective_bytes_per_value(bits, group, scale_bytes=2): bits/8 + scale_bytes/group (one scale shared by `group` values). group must be at least 1.
kv_bytes_per_token(values_per_token, bits, group, scale_bytes=2): values_per_token · effective_bytes_per_value.
block_round(tokens, block): the token slots allocated when memory is handed out in blocks: round up to a multiple of block.
allocated_bytes(tokens, block, per_token): block_round(tokens, block) · per_token.
overhead_fraction(tokens, block): wasted slots / allocated slots.

import math

def effective_bytes_per_value(bits, group, scale_bytes=2):
    pass

def kv_bytes_per_token(values_per_token, bits, group, scale_bytes=2):
    pass

def block_round(tokens, block):
    pass

def allocated_bytes(tokens, block, per_token):
    pass

def overhead_fraction(tokens, block):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is KV memory the whole GPU memory footprint?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a uniform full-attention decoder:

M_{KV}=2L H_{KV}d_h T B b.

L is layers, H_KV KV heads, d_h head dimension, T retained tokens per sequence, B equal-length unshared sequences and b bytes per value. The factor 2 counts keys and values. For unequal lengths, replace TB by the sum of retained lengths.

Worked example

For L=32, H_KV=8, d_h=128 and b=2, each retained token needs 2×32×8×128×2=131,072 bytes, or 128 KiB. At T=4096, one sequence needs 512 MiB. Four equally long independent sequences need 2 GiB, before allocator overhead and other memory.

Reference code

The following example uses the Python standard library and does not call a remote service.

layers,heads,dim,bytes_per_value=32,8,128,2
per_token=2*layers*heads*dim*bytes_per_value
assert per_token == 131072
assert per_token/1024 == 128
assert per_token*4096/1024**2 == 512
assert per_token*4096*4/1024**3 == 2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Use KV heads rather than query heads. Distinguish decimal GB from binary GiB. Hybrid, sliding-window and quantized-cache architectures require their own detailed accounting.