E6.6 · LESSON 6 OF 6

Quality and runtime

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

Quality and runtime together

A quantized model must be judged on accuracy, memory and speed, using kernels and workloads like the ones you will run. A fourfold smaller weight file does not give a fourfold faster model: if decoding is bandwidth-bound the weight reads shrink 4×, but unpacking codes, unsupported operators or dequantization overhead add time back. In the toy numbers 14 ms becomes 3.5 ms in memory time, but with 1.5 ms of unpacking the token takes 5 ms: a 2.8× speed-up.

weight bits4unpacking overhead (ms/token)1.5bandwidth (TB/s)1

FP16

14 ms

4-bit

5 ms

memory shrinks 4× but token latency improves 2.8× ideal (no overhead) would be 4×. Unsupported operators or slow unpacking can erase the gain.

Judge a quantized model on accuracy, memory and speed together, with kernels and workloads that match how you will run it.

Change the bit width, bandwidth and unpacking overhead.

Predict first. Is a fourfold weight-memory reduction automatically a fourfold latency improvement?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out a realistic speed-up by hand, then write the runtime model and a quality-budget chooser.

weight bits4unpacking overhead (ms/token)1.5bandwidth (TB/s)1

FP16

14 ms

4-bit

5 ms

memory shrinks 4× but token latency improves 2.8× ideal (no overhead) would be 4×. Unsupported operators or slow unpacking can erase the gain.

Judge a quantized model on accuracy, memory and speed together, with kernels and workloads that match how you will run it.

FP16 ms per token 4-bit ms, memory only with 1.5 ms unpacking speed-up

To do: Decoding is bandwidth-bound: time per token = bytes of weights / bandwidth. 7·10⁹ weights in FP16 (14 GB) at 1 TB/s: ms per token?
To do: At 4 bits (3.5 GB, ignoring scales) the memory time falls to…
To do: But unpacking the 4-bit codes to a usable format costs an extra 1.5 ms per token in this kernel. Total ms per token?
To do: Latency improvement over FP16 (ratio, 1 decimal)? The memory shrank 4×, but the speed-up is…

Part 2. Do it in code

memory_time_ms(n_params, bits, bandwidth_bytes_per_s): ms to read the weights once: n_params · bits / 8 / bandwidth · 1000.
token_time_ms(n_params, bits, bandwidth_bytes_per_s, overhead_ms=0.0): memory time plus overhead (for example unpacking the codes).
speedup(n_params, bits_base, bits_new, bandwidth_bytes_per_s, overhead_ms=0.0): token_time(base, no overhead) / token_time(new, with overhead).
memory_ratio(bits_base, bits_new): bits_base / bits_new.
acceptable(base_score, new_score, max_drop): True if the new score is at most max_drop below the baseline (higher scores are better).
pick_config(configs, base_score, max_drop): configs is a list of dicts with name, bits and score. Among those that are acceptable, return the name with the fewest bits (the first on ties); None if none is acceptable.

def memory_time_ms(n_params, bits, bandwidth_bytes_per_s):
    pass

def token_time_ms(n_params, bits, bandwidth_bytes_per_s, overhead_ms=0.0):
    pass

def speedup(n_params, bits_base, bits_new, bandwidth_bytes_per_s, overhead_ms=0.0):
    pass

def memory_ratio(bits_base, bits_new):
    pass

def acceptable(base_score, new_score, max_drop):
    pass

def pick_config(configs, base_score, max_drop):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is a fourfold weight-memory reduction automatically a fourfold latency improvement?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.