E6.1 · LESSON 1 OF 6

Floating formats

Estimate weight memory at different precisions and explain quantization error, scales and calibration.

THE IDEA

Number formats

Model weights are stored as numbers, and the format decides memory, range and detail. A floating-point number splits its bits into a sign, an exponent (the range) and a mantissa (the detail). FP16 uses 5 exponent and 10 mantissa bits; BF16 uses 8 and 7: the same 16 bits, but BF16 has the range of 32-bit floats and coarser detail. FP8 formats exist in more than one layout, so a bit count alone does not specify the arithmetic.

FP16

sign 1exponent 5mantissa 10

BF16

sign 1exponent 8mantissa 7

	FP16	BF16
exponent bits	5	8
mantissa bits	10	7
largest finite value	65,504	3.390e+38
gap between neighbours near 1	2⁻¹⁰ = 0.0009766	2⁻⁷ = 0.0078125

value near 1 (offset ×2⁻¹⁰)1.001953125

x = 1.001953125 stored as FP16: 1.001953125 stored as BF16: 1 Same width, different trade: FP16 keeps finer detail, BF16 keeps a much wider range.

A bit count alone does not define an arithmetic contract: the exponent/mantissa split, rounding and special values matter. FP8 comes in more than one layout too.

Pick a value near 1 and compare how each format stores it.

Predict first. Are FP16 and BF16 numerically identical because both use 16 bits?

Chapter context

Quantization represents values with a smaller set of numerical codes. Memory savings are easy to estimate nominally, but quality and speed depend on scales, calibration, kernels and which tensors use the smaller representation.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out the bit layouts and a rounding case by hand, then compute format limits and write a real BF16 rounding function.

FP16

sign 1exponent 5mantissa 10

BF16

sign 1exponent 8mantissa 7

	FP16	BF16
exponent bits	5	8
mantissa bits	10	7
largest finite value	65,504	3.390e+38
gap between neighbours near 1	2⁻¹⁰ = 0.0009766	2⁻⁷ = 0.0078125

value near 1 (offset ×2⁻¹⁰)1.001953125

x = 1.001953125 stored as FP16: 1.001953125 stored as BF16: 1 Same width, different trade: FP16 keeps finer detail, BF16 keeps a much wider range.

A bit count alone does not define an arithmetic contract: the exponent/mantissa split, rounding and special values matter. FP8 comes in more than one layout too.

FP16 mantissa bits BF16 mantissa bits largest FP16 BF16 value nearest 1.001953125

To do: FP16 splits its 16 bits into 1 sign bit, 5 exponent bits and the rest for the fraction (mantissa). How many mantissa bits?
To do: BF16 also has 16 bits and 1 sign bit, but 8 exponent bits (the same range as 32-bit floats). How many mantissa bits does it have?
To do: The largest finite FP16 number is (2 − 2⁻¹⁰) · 2¹⁵. Type its value
To do: The number 1 + 2⁻⁹ = 1.001953125 fits in FP16 exactly. The gap between neighbouring BF16 numbers near 1 is 2⁻⁷ = 0.0078125. Which BF16 value is nearest to 1.001953125? Type it

Part 2. Do it in code

FORMATS maps a name to (exponent_bits, mantissa_bits): "fp16" (5, 10), "bf16" (8, 7), "fp32" (8, 23). Define it yourself.
total_bits(name): 1 sign bit + exponent bits + mantissa bits.
max_finite(name): (2 − 2^−m) · 2^(2^(e−1) − 1) for exponent bits e and mantissa bits m.
relative_spacing(name): 2^−m, the gap between neighbouring numbers near 1.
to_bf16(x): round a number to the nearest BF16 value (round-to-nearest-even on the upper 16 bits of the float32 pattern). Use struct to get the 32-bit pattern.
to_fp16(x): the value after storing it as np.float16 (infinity if it overflows).

import struct
import numpy as np

FORMATS = {}

def total_bits(name):
    pass

def max_finite(name):
    pass

def relative_spacing(name):
    pass

def to_bf16(x):
    pass

def to_fp16(x):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Are FP16 and BF16 numerically identical because both use 16 bits?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Nominal packed weight memory is

M=N\,b/8\;\text{bytes}.

For affine quantization, $q=\operatorname{clip}(\operatorname{round}(x/s)+z)$ and $\hat x=s(q-z)$ . Here s is scale, z zero point and clipping uses the chosen integer range. Metadata and mixed-precision tensors add storage.

Worked example

Seven billion weights at 16 bits occupy 14 billion bytes (14 decimal GB) nominally. At 4 bits they occupy 3.5 decimal GB, a fourfold nominal reduction. Separately, x=0.23 with s=0.1,z=0 rounds to q=2 and reconstructs 0.2, an absolute error of 0.03.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
count=7_000_000_000
assert count*16/8 == 14_000_000_000
assert count*4/8 == 3_500_000_000
q=round(.23/.1)
assert q == 2
assert isclose(abs(.23-.1*q),.03)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Nominal bit packing excludes scales, padding and other state. Different 16-bit float formats have different ranges and precision. Quantization quality and kernel speed both need measurement.