E6.2 · LESSON 2 OF 6

Integer quantization

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

Integer quantization: scale and zero point

Integer quantization stores a small integer code instead of a float. A scale (and optionally a zero point) maps between them: code = round(x / scale) + zero, and x ≈ (code − zero) · scale. Two errors appear. Rounding error is at most half a step. Clipping error appears when x is too big for the range and is cut to the largest code.

The worked example: x = 0.23, scale 0.1, zero point 0 gives code 2 (2.3 rounds to 2), reconstructs 0.2, error 0.03.

value x0.23scale0.1bits8

code = clip(round(x / scale), -128, 127) = clip(round(2.3)) = 2 reconstruction = code × scale = 0.2 absolute error = 0.03 rounding error is at most scale/2 = 0.05; clipping error can be much larger

Blue ticks: the values the codes can represent. Orange: your x. Green: what it becomes.

Move x, the scale and the bit width. Try x = 15 with 8 bits.

Predict first. With scale 0.1 and zero point zero, what code represents 0.23 after rounding?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Quantize and reconstruct a few values by hand, then write quantize, dequantize and the scale for a range.

value x0.23scale0.1bits8

code = clip(round(x / scale), -128, 127) = clip(round(2.3)) = 2 reconstruction = code × scale = 0.2 absolute error = 0.03 rounding error is at most scale/2 = 0.05; clipping error can be much larger

Blue ticks: the values the codes can represent. Orange: your x. Green: what it becomes.

code for 0.23 at scale 0.1 reconstructed value 15 at scale 0.1 in int8 reconstructs to

To do: Scale 0.1, zero point 0. The integer code for x = 0.23 is round(x / scale). Type it
To do: The code is dequantized as code × scale. Reconstructed value (and the absolute error is |0.23 − that|)?
To do: An int8 range is −128…127. With scale 0.1 the value 15 would need code 150, so it is clipped to 127. What value does it reconstruct to?

Part 2. Do it in code

int_range(bits, signed=True): (lowest, highest) code, for example (−128, 127) for 8 signed bits.
quantize(x, scale, zero=0, bits=8, signed=True): codes = clip(round(x / scale) + zero, lowest, highest). Use np.round (round half to even). scale must be positive (ValueError). Works on arrays.
dequantize(q, scale, zero=0): (q − zero) · scale.
quant_error(x, scale, zero=0, bits=8): the absolute error |x − dequantize(quantize(x))|.
scale_for_range(lo, hi, bits=8): asymmetric parameters (scale, zero) so that lo maps to the lowest code and hi to the highest unsigned code: scale = (hi − lo) / (2^bits − 1), zero = round(−lo / scale). ValueError if hi <= lo.

import numpy as np

def int_range(bits, signed=True):
    pass

def quantize(x, scale, zero=0, bits=8, signed=True):
    pass

def dequantize(q, scale, zero=0):
    pass

def quant_error(x, scale, zero=0, bits=8):
    pass

def scale_for_range(lo, hi, bits=8):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

With scale .1 and zero point zero, what code represents .23 after rounding?

AThree, reconstructing 0.3.BTwo, reconstructing 0.2: 0.23 / 0.1 = 2.3 rounds to 2, an error of 0.03.CTwenty-three, reconstructing 2.3.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.