Latency measurements
Trace a generation request through prefill and decode and distinguish latency, throughput and hardware bottlenecks.
THE IDEA
Naming the latencies
"Latency" is not one number. TTFT is the time until the first output. Inter-token latency is the gap between consecutive outputs. End-to-end latency is the time until the last token. Say which events you start and stop the clock on.
Worked example: TTFT 0.20 s, then four more tokens at 0.05 s intervals. Five outputs finish at 0.20 + 4 × 0.05 = 0.40 s. The steady decode rate is 1/0.05 = 20 tokens/s, while the whole request produced 5/0.40 = 12.5 tokens/s. Both are valid, with different definitions.
Both rates are correct measurements of different things. Always state which events define a latency (request sent? first byte? first token shown?).
Predict first. For five outputs, how many gaps follow the first output?
Chapter context
An autoregressive model first processes a prompt, then generates tokens whose identities determine later steps. Prefill and decode often have different hardware behavior, so a single tokens-per-second number does not describe the whole request.
TRY IT YOURSELF
Part 1. Do it hands-on
Compute each latency measure for the example by hand, then write a metrics function from timestamps and a percentile for tail latency.
Both rates are correct measurements of different things. Always state which events define a latency (request sent? first byte? first token shown?).
- To do: A request has TTFT = 0.20 s and emits 5 output tokens in all. How many inter-token gaps follow the first token?
- To do: The gaps are 0.05 s each. End-to-end latency?
- To do: Steady decode rate = 1 / inter-token gap (tokens per second)?
- To do: Whole-request output rate = tokens / end-to-end time (5 tokens in 0.4 s)?
Part 2. Do it in code
- metrics(request_time, token_times): token_times are the clock times at which each output token appeared. Return a dict with ttft (first − request), gaps (list of differences between neighbouring outputs), e2e (last − request), mean_gap, steady_rate (1 / mean_gap, or None for fewer than 2 tokens) and overall_rate (tokens / e2e). ValueError if there are no tokens or the times are not increasing.
- percentile(values, q): the q-th percentile (0–100) with linear interpolation between sorted neighbours (like numpy's default). ValueError for an empty list.
- tail_latency(list_of_e2e, q): percentile of end-to-end times (a convenience for p95/p99).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
For five outputs, how many gaps follow the first output?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
For roughly steady output gaps:
For a simplified single-sequence dense weight scan, FLOPs≈2N and weight bytes≈Nb, giving arithmetic intensity≈2/b FLOPs per byte. This excludes activations, KV traffic and other work.
Worked example
A fictional request has TTFT=0.20 seconds and then emits four more tokens at 0.05-second intervals. Five outputs finish in 0.20+4×0.05=0.40 seconds. Steady decode rate is 1/0.05=20 tokens/second; total request output rate is 5/0.40=12.5 tokens/second. Both are valid measurements with different definitions.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import isclose
first,gap,count=.2,.05,5
elapsed=first+(count-1)*gap
assert isclose(elapsed,.4)
assert 1/gap == 20
assert isclose(count/elapsed,12.5)
assert 2/2 == 1
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
An average latency hides tail behavior. Prefill/decode bottlenecks depend on batch and shapes, not only phase names. Compare metric definitions before comparing serving engines.