E6.4 · LESSON 4 OF 6

GPTQ and AWQ

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

4-bit weights, and named methods

At 4 bits there are only 16 codes, so rounding hurts more. Two well-known post-training methods try to lose less quality. GPTQ quantizes weights one at a time, using approximate second-order information to adjust the remaining weights for the error already made. AWQ is activation-aware: it looks at which input channels see large activations and protects the corresponding weights. These are specific algorithms. "INT4" names only the code width.

signed 4-bit codes (−8 … 7)

byte	first code	second code	bits (second \| first)	byte value
0	1	-2	1110 \| 0001	225
1	7	-8	1000 \| 0111	135

4 codes → 2 bytes (two per byte). 7 billion weights → 3.5 billion bytes before scales. "INT4" says how wide the codes are; it does not say whether GPTQ, AWQ or plain rounding chose them.

GPTQ uses approximate second-order information to choose the rounded weights; AWQ uses activation statistics to protect the weights that matter most. Both are specific methods. This lesson's exercise shows the storage side and a weighted error that such methods try to reduce.

Packing signed 4-bit codes, two per byte.

Predict first. Does an INT4 label tell us which quantization algorithm was used?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute 4-bit sizes by hand, then pack and unpack real 4-bit codes and compute an importance-weighted error.

signed 4-bit codes (−8 … 7)

byte	first code	second code	bits (second \| first)	byte value
0	1	-2	1110 \| 0001	225
1	7	-8	1000 \| 0111	135

4 codes → 2 bytes (two per byte). 7 billion weights → 3.5 billion bytes before scales. "INT4" says how wide the codes are; it does not say whether GPTQ, AWQ or plain rounding chose them.

GB for 7e9 weights at 4 bits values a 4-bit code can take largest signed 4-bit code does INT4 name the algorithm? (yes/no)

To do: 7 billion weights at exactly 4 bits each (ignoring scales). Decimal GB?
To do: How many different values can a 4-bit code take?
To do: A SIGNED 4-bit integer covers −8 … what is the largest code?
To do: A file is labelled "INT4". Does that label tell you whether GPTQ, AWQ or plain rounding produced it? Type yes or no

Part 2. Do it in code

pack_int4(codes): codes are integers in −8…7. Store two per byte: byte = (low & 0xF) | ((high & 0xF) << 4), where low is the first code of the pair and high the second. An odd number of codes is padded with 0. Return a list of byte values (0–255). ValueError for an out-of-range code.
unpack_int4(data, n): the first n signed codes back (sign-extend each 4-bit field).
packed_size(n_codes): bytes needed for n_codes 4-bit codes.
rtn(w, scale): plain round-to-nearest 4-bit quantization; return the dequantized values clip(round(w/scale), −8, 7)·scale (np.round, ties to even).
weighted_error(w, w_hat, importance): sum(importance · (w − w_hat)²). Activation-aware methods try to make this small for the weights that matter most.

import numpy as np

def pack_int4(codes):
    pass

def unpack_int4(data, n):
    pass

def packed_size(n_codes):
    pass

def rtn(w, scale):
    pass

def weighted_error(w, w_hat, importance):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does an INT4 label tell us which quantization algorithm was used?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.