E6.3 · LESSON 3 OF 6

Granularity

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

How many scales?

One scale for a whole tensor is cheap but must cover its largest weight, which makes small weights coarse. One scale per channel (row) or per group of a few weights fits local ranges better, at the cost of more scale numbers to store and handle. Always state which axis a scale describes.

row	scale	weights	reconstructed (int8)	mean \|error\|
A (small)	0.07874	0.1, 0.2, -0.1, 0.15	0.0787, 0.2362, -0.0787, 0.1575	0.02156
B (large)	0.07874	10, -8, 9, 7	10, -8.0315, 8.9764, 7.0079	0.01575

average error over all 8 weights: 0.01865 (per-tensor 0.01865, per-row 0.00812) metadata: 1 scale (a 4×8 matrix: 1 per tensor, 4 per channel, 8 per group of 4)

With one scale sized for the big row, the small row is quantized very coarsely. Finer granularity fits local ranges better and costs more scale metadata; always state which axis a scale describes.

A row of small weights next to a row of large ones.

Predict first. Does one scale per channel have the same metadata count as one scale per tensor?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Count scales and compare errors by hand, then quantize a matrix at three granularities.

row	scale	weights	reconstructed (int8)	mean \|error\|
A (small)	0.07874	0.1, 0.2, -0.1, 0.15	0.0787, 0.2362, -0.0787, 0.1575	0.02156
B (large)	0.07874	10, -8, 9, 7	10, -8.0315, 8.9764, 7.0079	0.01575

average error over all 8 weights: 0.01865 (per-tensor 0.01865, per-row 0.00812) metadata: 1 scale (a 4×8 matrix: 1 per tensor, 4 per channel, 8 per group of 4)

With one scale sized for the big row, the small row is quantized very coarsely. Finer granularity fits local ranges better and costs more scale metadata; always state which axis a scale describes.

scales per tensor per channel (4 rows) groups of 4 over 32 weights smaller error: tensor or channel

To do: A weight matrix has 4 rows (output channels) and 8 columns. How many scales with one scale per TENSOR?
To do: One scale per output channel (per row)?
To do: One scale per group of 4 consecutive weights, over all 32 weights?
To do: Row A holds small weights (0.1, 0.2, −0.1, 0.15), row B large ones (10, −8, 9, 7). Which scale choice gives the smaller average error for 8-bit codes: tensor or channel? Type the word

Part 2. Do it in code

sym_scale(values, bits=8): max|values| / (2^(bits−1) − 1). A tensor of zeros gets scale 1.0.
quant_dequant(w, scale): symmetric int8 round trip: clip(round(w/scale), −128, 127) · scale.
scale_count(shape, mode, group=None): number of scales for a 2-D weight of that shape: "tensor" → 1, "channel" (one per row) → rows, "group" (groups of `group` consecutive weights within each row; group must divide the row length, ValueError otherwise) → rows · cols / group.
quantize_matrix(W, mode, group=None): return the dequantized matrix when each tensor/row/group has its own symmetric scale.
mean_abs_error(W, mode, group=None): mean |W − quantize_matrix(W, mode, group)|.

import numpy as np

def sym_scale(values, bits=8):
    pass

def quant_dequant(w, scale):
    pass

def scale_count(shape, mode, group=None):
    pass

def quantize_matrix(W, mode, group=None):
    pass

def mean_abs_error(W, mode, group=None):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does one scale per channel have the same metadata count as one scale per tensor?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.