E1.3 · LESSON 3 OF 6

Decode

Trace a generation request through prefill and decode and distinguish latency, throughput and hardware bottlenecks.

THE IDEA

Decode: one token at a time

After prefill, the model generates. Each new token is chosen from a distribution that depends on every earlier token, including the one just generated. So the steps cannot be done in parallel along the time axis: token 2 cannot be known before token 1 is picked.

Each step also reads every weight from memory to do a small amount of arithmetic for one new position. When that memory traffic is the limit, the time per token is roughly (bytes of weights) / (memory bandwidth): 14 GB at 1 TB/s is 14 ms, about 71 tokens per second at best for one sequence.

parameters (billions)7bytes per parameter2bandwidth (TB/s)1

weights read per step = 7e9 × 2 bytes = 14 GB time per token ≥ 14 GB / 1 TB/s = 14 ms → at most 71.4 tokens/s for one sequence

Token 1 of 5

The????

Step 1: the model reads all weights, uses the 0 token(s) already produced, and only then can it pick token 1. It cannot know token 2 first.

Step through generation, and change the model size, precision and bandwidth.

Predict first. Can an ordinary autoregressive decode know the second generated token before selecting the first?

Chapter context

An autoregressive model first processes a prompt, then generates tokens whose identities determine later steps. Prefill and decode often have different hardware behavior, so a single tokens-per-second number does not describe the whole request.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute a bandwidth-bound decode speed by hand, then model the step time and prove generation is sequential.

parameters (billions)7bytes per parameter2bandwidth (TB/s)1

weights read per step = 7e9 × 2 bytes = 14 GB time per token ≥ 14 GB / 1 TB/s = 14 ms → at most 71.4 tokens/s for one sequence

Token 1 of 5

The????

Step 1: the model reads all weights, uses the 0 token(s) already produced, and only then can it pick token 1. It cannot know token 2 first.

GB read per step (7e9 params × 2 bytes) ms per token at 1 TB/s tokens/s (1 decimal)

To do: Each decode step for one sequence must read all weights once. A 7·10⁹-parameter model in 2-byte numbers: how many GB per step?
To do: Memory bandwidth is 1 TB/s (10¹² bytes/s). Milliseconds per decoded token at best?
To do: So a single sequence decodes at most about how many tokens per second? (1 decimal)

Part 2. Do it in code

decode_step_seconds(param_bytes, bandwidth): time to read all weights once: param_bytes / bandwidth.
tokens_per_second(param_bytes, bandwidth): the best single-sequence decode rate, 1 / decode_step_seconds.
decode_time(n_tokens, param_bytes, bandwidth): total time to generate n_tokens one after another.
generate(next_token, prompt, n): next_token(sequence) returns the next id. Append n tokens, calling it once per token on the sequence so far. Return the full list.
call_order_is_sequential(next_token, prompt, n): wrap next_token to record the length of the sequence at every call; return True if the lengths are len(prompt), len(prompt)+1, ... (each call needs the previous result).

def decode_step_seconds(param_bytes, bandwidth):
    pass

def tokens_per_second(param_bytes, bandwidth):
    pass

def decode_time(n_tokens, param_bytes, bandwidth):
    pass

def generate(next_token, prompt, n):
    pass

def call_order_is_sequential(next_token, prompt, n):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can an ordinary autoregressive decode know the second generated token before selecting the first?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For roughly steady output gaps:

T_{request}\approx TTFT+(n_{out}-1)\,ITL.

For a simplified single-sequence dense weight scan, FLOPs≈2N and weight bytes≈Nb, giving arithmetic intensity≈2/b FLOPs per byte. This excludes activations, KV traffic and other work.

Worked example

A fictional request has TTFT=0.20 seconds and then emits four more tokens at 0.05-second intervals. Five outputs finish in 0.20+4×0.05=0.40 seconds. Steady decode rate is 1/0.05=20 tokens/second; total request output rate is 5/0.40=12.5 tokens/second. Both are valid measurements with different definitions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
first,gap,count=.2,.05,5
elapsed=first+(count-1)*gap
assert isclose(elapsed,.4)
assert 1/gap == 20
assert isclose(count/elapsed,12.5)
assert 2/2 == 1

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An average latency hides tail behavior. Prefill/decode bottlenecks depend on batch and shapes, not only phase names. Compare metric definitions before comparing serving engines.