E1.6 · LESSON 6 OF 6

Compute and bandwidth

Trace a generation request through prefill and decode and distinguish latency, throughput and hardware bottlenecks.

THE IDEA

Compute-bound or bandwidth-bound?

Arithmetic intensity is the number of operations done per byte moved from memory. A chip can do at most min(peak compute, bandwidth × intensity) operations per second (the "roofline"). Below the ridge point (peak / bandwidth) a step is limited by memory traffic.

Decoding one sequence multiplies a weight matrix by a single vector: each byte of weights is used for only a couple of operations, so intensity is low and the step is bandwidth-bound. Batching many sequences reuses each weight for every sequence, which raises the intensity.

batch size b1peak compute, log10 FLOP/s10^14

intensity = 2·b / 2 bytes = 1 FLOPs per byte attainable = min(1.0e+14, 1e12 × 1) = 1.00e+12 FLOP/s (1% of peak) ridge point = peak / bandwidth = 100 FLOPs per byte → BANDWIDTH-bound: more peak FLOPs would not help

Batch-1 decode multiplies every weight by a single vector, so each byte read does very little arithmetic. Larger batches reuse each weight for more vectors and move toward the compute limit. Raise the peak slider with b = 1: the attainable rate does not move.

Move the batch size and the peak compute.

Predict first. Does more peak FLOPs guarantee faster bandwidth-bound decoding?

Chapter context

An autoregressive model first processes a prompt, then generates tokens whose identities determine later steps. Prefill and decode often have different hardware behavior, so a single tokens-per-second number does not describe the whole request.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out intensity and the roofline by hand, then code the roofline model and find the batch size that saturates the chip.

batch size b1peak compute, log10 FLOP/s10^14

intensity at b = 1 attainable FLOP/s at intensity 1 ridge point (FLOPs/byte)

To do: A weight matrix is read once per step. With batch size b, FLOPs ≈ 2·b per weight and the weight costs 2 bytes, so arithmetic intensity = b FLOPs per byte. For b = 1?
To do: Peak compute 10¹⁴ FLOP/s, bandwidth 10¹² byte/s. At intensity 1, attainable FLOP/s = min(peak, bandwidth × intensity). Type it (like 1e12)
To do: What intensity (FLOPs per byte) is needed before the machine stops being bandwidth-bound? peak / bandwidth

Part 2. Do it in code

arithmetic_intensity(flops, bytes_moved): flops / bytes_moved (ValueError if bytes_moved <= 0).
matmul_intensity(b, bytes_per_weight=2): intensity of a weight matrix used by a batch of b vectors when the weights dominate the traffic: 2·b / bytes_per_weight.
attainable(peak_flops, bandwidth, intensity): min(peak_flops, bandwidth · intensity).
ridge_point(peak_flops, bandwidth): peak_flops / bandwidth.
bound(peak_flops, bandwidth, intensity): the string "bandwidth" if intensity < ridge_point, otherwise "compute".
batch_to_saturate(peak_flops, bandwidth, bytes_per_weight=2): the smallest integer batch size whose matmul_intensity reaches the ridge point.

import math

def arithmetic_intensity(flops, bytes_moved):
    pass

def matmul_intensity(b, bytes_per_weight=2):
    pass

def attainable(peak_flops, bandwidth, intensity):
    pass

def ridge_point(peak_flops, bandwidth):
    pass

def bound(peak_flops, bandwidth, intensity):
    pass

def batch_to_saturate(peak_flops, bandwidth, bytes_per_weight=2):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does more peak FLOPs guarantee faster bandwidth-bound decoding?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For roughly steady output gaps:

T_{request}\approx TTFT+(n_{out}-1)\,ITL.

For a simplified single-sequence dense weight scan, FLOPs≈2N and weight bytes≈Nb, giving arithmetic intensity≈2/b FLOPs per byte. This excludes activations, KV traffic and other work.

Worked example

A fictional request has TTFT=0.20 seconds and then emits four more tokens at 0.05-second intervals. Five outputs finish in 0.20+4×0.05=0.40 seconds. Steady decode rate is 1/0.05=20 tokens/second; total request output rate is 5/0.40=12.5 tokens/second. Both are valid measurements with different definitions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
first,gap,count=.2,.05,5
elapsed=first+(count-1)*gap
assert isclose(elapsed,.4)
assert 1/gap == 20
assert isclose(count/elapsed,12.5)
assert 2/2 == 1

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An average latency hides tail behavior. Prefill/decode bottlenecks depend on batch and shapes, not only phase names. Compare metric definitions before comparing serving engines.