E1.1 · LESSON 1 OF 6

The request path

Trace a generation request through prefill and decode and distinguish latency, throughput and hardware bottlenecks.

THE IDEA

Where the time goes before the first word

A user sends a prompt and waits. Between pressing send and seeing the first token, the request may wait in a queue, travel over the network, be processed by the model (the prompt phase), and produce its first token. Client-side rendering adds a little more. The measured "time to first token" (TTFT) is the sum of all of these.

waiting in the queue0.05 snetwork0.02 sprefill (model work on the prompt)0.12 sfirst decode step0.01 s

queue

prefill

time to first token = 0.05 + 0.02 + 0.12 + 0.01 = 0.2 s queueing share: 25% GPU time alone would not explain the observed number.

Drag the components. Notice that a traffic spike changes TTFT without any change to the model.

Predict first. Does time to first token necessarily equal GPU compute time?

Chapter context

An autoregressive model first processes a prompt, then generates tokens whose identities determine later steps. Prefill and decode often have different hardware behavior, so a single tokens-per-second number does not describe the whole request.

TRY IT YOURSELF

Part 1. Do it hands-on

Add up a request's path, see the effect of a queue spike, then write TTFT, breakdown and bottleneck functions.

waiting in the queue0.05 snetwork0.02 sprefill (model work on the prompt)0.12 sfirst decode step0.01 s

queue

prefill

time to first token = 0.05 + 0.02 + 0.12 + 0.01 = 0.2 s queueing share: 25% GPU time alone would not explain the observed number.

TTFT for the defaults queueing share TTFT with a 0.30 s queue

To do: Time to first token adds everything before the first token appears: queue 0.05 s + network 0.02 s + prefill 0.12 s + first decode step 0.01 s. Total?
To do: What fraction of that TTFT is queueing? (as a decimal)
To do: The GPU work is unchanged but a traffic spike pushes the queue wait to 0.30 s. New TTFT?

Part 2. Do it in code

ttft(parts): the sum of the values of a dict of time components (queue, network, prefill, ...). ValueError for a negative component.
breakdown(parts): a dict with each component's share of the total (the shares add up to 1). ValueError if the total is 0.
bottleneck(parts): the name of the largest component (the alphabetically first on ties).
meets_slo(parts, limit): True if ttft(parts) is at most limit.

def ttft(parts):
    pass

def breakdown(parts):
    pass

def bottleneck(parts):
    pass

def meets_slo(parts, limit):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does time to first token necessarily equal GPU compute time?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For roughly steady output gaps:

T_{request}\approx TTFT+(n_{out}-1)\,ITL.

For a simplified single-sequence dense weight scan, FLOPs≈2N and weight bytes≈Nb, giving arithmetic intensity≈2/b FLOPs per byte. This excludes activations, KV traffic and other work.

Worked example

A fictional request has TTFT=0.20 seconds and then emits four more tokens at 0.05-second intervals. Five outputs finish in 0.20+4×0.05=0.40 seconds. Steady decode rate is 1/0.05=20 tokens/second; total request output rate is 5/0.40=12.5 tokens/second. Both are valid measurements with different definitions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
first,gap,count=.2,.05,5
elapsed=first+(count-1)*gap
assert isclose(elapsed,.4)
assert 1/gap == 20
assert isclose(count/elapsed,12.5)
assert 2/2 == 1

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An average latency hides tail behavior. Prefill/decode bottlenecks depend on batch and shapes, not only phase names. Compare metric definitions before comparing serving engines.