E6.5 · LESSON 5 OF 6

Containers and KV state

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

Containers and the KV cache

A quantized model has to be stored somewhere. GGUF is one file format; it holds tensors with associated encodings and metadata, and different tensors may use different encodings, so it is a container, not a precision. Scales are part of the real size: 4-bit codes with a 2-byte scale per 64 weights cost 0.53125 bytes per weight, so 7 billion weights take 3.72 GB, not 3.5.

The KV cache is separate state. Quantizing the weights does not change it: it stays in its own format (often 16-bit) unless you also choose to quantize it.

weight bits4values per scale64KV bytes per value2tokens in cache4,096

weights

weights: 7e9 × (4/8 + 2/64) = 3.719 GB KV cache (reference model, 4,096 tokens, 2 byte(s) per value): 0.537 GB total ≈ 4.256 GB

Weight quantization and KV-cache quantization are separate choices on separate state. A model file format such as GGUF is a container that can hold tensors in several encodings; it is not one precision.

Weights and a KV cache are separate memory items.

Predict first. Does quantizing model weights automatically quantize the KV cache?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Add up weights with scales and a KV cache by hand, then write the memory functions.

weight bits4values per scale64KV bytes per value2tokens in cache4,096

weights

weights: 7e9 × (4/8 + 2/64) = 3.719 GB KV cache (reference model, 4,096 tokens, 2 byte(s) per value): 0.537 GB total ≈ 4.256 GB

GB for 7e9 weights, 4 bits, scale per 64 (3 decimals) FP16 KV bytes for 4096 tokens total GB (3 decimals)

To do: 7·10⁹ weights stored as 4-bit codes with a 2-byte scale per group of 64: bytes per weight = 0.5 + 2/64. Decimal GB (3 decimals)?
To do: The KV cache is separate state. If it stays in FP16, the reference model needs 512 MiB for one 4096-token sequence. In bytes (1 MiB = 1,048,576)?
To do: Weights + one 4096-token KV cache, in decimal GB (3 decimals)?

Part 2. Do it in code

weight_bytes(n_params, bits, group=0, scale_bytes=2): n_params · (bits/8 + scale_bytes/group), with no scale term if group is 0.
kv_bytes(layers, kv_heads, head_dim, bytes_per_value, tokens): 2 · layers · kv_heads · head_dim · bytes_per_value · tokens.
total_gb(weights_bytes, kv_bytes_value): (weights + kv) / 1e9.
kv_changes_with_weight_quant(weight_bits, kv_bytes_per_value): return False: quantizing the weights does not change the KV cache, which is determined by kv_bytes_per_value alone; but return True if kv_bytes_per_value is smaller than 2 (the KV cache itself was quantized).
container_summary(name, weight_bits, group, kv_bytes_per_value): a dict with name, weight_bits, group and kv_bytes_per_value. (A model file such as GGUF can hold tensors of different precisions; it is a container, not one precision.)

def weight_bytes(n_params, bits, group=0, scale_bytes=2):
    pass

def kv_bytes(layers, kv_heads, head_dim, bytes_per_value, tokens):
    pass

def total_gb(weights_bytes, kv_bytes_value):
    pass

def kv_changes_with_weight_quant(weight_bits, kv_bytes_per_value):
    pass

def container_summary(name, weight_bits, group, kv_bytes_per_value):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does quantizing model weights automatically quantize the KV cache?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.