E1.5 · LESSON 5 OF 6

Throughput and goodput

Trace a generation request through prefill and decode and distinguish latency, throughput and hardware bottlenecks.

THE IDEA

Throughput versus goodput

Throughput is completed work per unit time over a workload, for example tokens per second summed over all requests. Goodput counts only the work that meets a stated service objective. Putting more requests in one batch usually raises the total, because the weights are read once for all of them, but each request then gets its tokens more slowly.

per-request objective (tokens/s)10

batch	step (s)	aggregate tokens/s	each request	meets objective?
1	0.05	20	20	yes
2	0.06	33.3	16.7	yes
3	0.07	42.9	14.3	yes
4	0.08	50	12.5	yes
5	0.09	55.6	11.1	yes
6	0.1	60	10	yes
7	0.11	63.6	9.1	no
8	0.12	66.7	8.3	no
9	0.13	69.2	7.7	no
10	0.14	71.4	7.1	no

step time = 0.05 + 0.01·(b − 1) s (a toy model) throughput counts all tokens/s; goodput counts only the work that meets the objective (here: batches where every request gets ≥ the objective).

A toy model where each extra sequence adds 10 ms to a step. Change the per-request objective.

Predict first. Can batching raise aggregate throughput while delaying one request?

Chapter context

An autoregressive model first processes a prompt, then generates tokens whose identities determine later steps. Prefill and decode often have different hardware behavior, so a single tokens-per-second number does not describe the whole request.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute total and per-request speed for a batch, pick the best batch under an objective, then code the toy model and goodput.

per-request objective (tokens/s)10your batch size1

batch	step (s)	aggregate tokens/s	each request	meets objective?
1	0.05	20	20	yes
2	0.06	33.3	16.7	yes
3	0.07	42.9	14.3	yes
4	0.08	50	12.5	yes
5	0.09	55.6	11.1	yes
6	0.1	60	10	yes
7	0.11	63.6	9.1	no
8	0.12	66.7	8.3	no
9	0.13	69.2	7.7	no
10	0.14	71.4	7.1	no

step time = 0.05 + 0.01·(b − 1) s (a toy model) throughput counts all tokens/s; goodput counts only the work that meets the objective (here: batches where every request gets ≥ the objective).

aggregate for b = 5 per request for b = 5 Use the batch slider above for the third task (objective 10).

To do: Model: a step takes 0.05 + 0.01 · (b − 1) seconds for b sequences batched, and each step gives every sequence one token. For b = 5 the step takes 0.09 s. Aggregate throughput in tokens/s (1 decimal)?
To do: Each of those 5 requests receives how many tokens per second? (1 decimal)
To do: Service objective: every request must get at least 10 tokens/s. Choose the batch size that gives the HIGHEST aggregate throughput while meeting it

Part 2. Do it in code

step_time(b, base=0.05, per_extra=0.01): base + per_extra · (b − 1) for a batch of b sequences. b must be at least 1.
aggregate_tps(b) = b / step_time(b); per_request_tps(b) = 1 / step_time(b).
goodput(b, slo_tps): the aggregate tokens/s if every request in the batch meets the per-request objective (per_request_tps(b) >= slo_tps), otherwise 0.
best_batch(slo_tps, max_b=64): the batch size with the highest goodput (the smallest on ties); 0 if no batch size meets the objective.

def step_time(b, base=0.05, per_extra=0.01):
    pass

def aggregate_tps(b):
    pass

def per_request_tps(b):
    pass

def goodput(b, slo_tps):
    pass

def best_batch(slo_tps, max_b=64):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can batching raise aggregate throughput while delaying one request?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For roughly steady output gaps:

T_{request}\approx TTFT+(n_{out}-1)\,ITL.

For a simplified single-sequence dense weight scan, FLOPs≈2N and weight bytes≈Nb, giving arithmetic intensity≈2/b FLOPs per byte. This excludes activations, KV traffic and other work.

Worked example

A fictional request has TTFT=0.20 seconds and then emits four more tokens at 0.05-second intervals. Five outputs finish in 0.20+4×0.05=0.40 seconds. Steady decode rate is 1/0.05=20 tokens/second; total request output rate is 5/0.40=12.5 tokens/second. Both are valid measurements with different definitions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
first,gap,count=.2,.05,5
elapsed=first+(count-1)*gap
assert isclose(elapsed,.4)
assert 1/gap == 20
assert isclose(count/elapsed,12.5)
assert 2/2 == 1

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An average latency hides tail behavior. Prefill/decode bottlenecks depend on batch and shapes, not only phase names. Compare metric definitions before comparing serving engines.