E9.5 · LESSON 5 OF 6

Observability

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

Measuring what users feel

Track the stages of a request with consistent definitions: queueing time, time to first token, the gaps between tokens, failures and end-to-end latency. Record them as histograms so you can read percentiles. The mean can hide the slow tail: with 95 requests at 0.2 s and 5 at 3.0 s the mean is 0.34 s, yet 5% of users waited three seconds.

requests (of 100) taking 3.0 s instead of 0.2 s5

bucket	count
< 0.5 s	95
0.5 – 1 s	0
1 – 5 s	5

mean = 0.34 s p50 = 0.2 s p95 = 0.34 s p99 = 3 s fraction slower than 1 s = 0.05

Measure queueing, time to first token, the gaps between tokens, failures and end-to-end time with consistent definitions. Histograms show the shape; a mean can look healthy while users in the tail wait seconds.

Increase the number of slow requests.

Predict first. Does mean latency alone establish a p95 target?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Compare the mean with the tail by hand, then write percentile, histogram and timing helpers.

requests (of 100) taking 3.0 s instead of 0.2 s5

bucket	count
< 0.5 s	95
0.5 – 1 s	0
1 – 5 s	5

mean = 0.34 s p50 = 0.2 s p95 = 0.34 s p99 = 3 s fraction slower than 1 s = 0.05

mean (95 × 0.2 s, 5 × 3.0 s) p99 (1 decimal) fraction over 1 s

To do: 100 requests: 95 take 0.2 s and 5 take 3.0 s. Mean latency (seconds, 2 decimals)?
To do: The 99th percentile (linear interpolation on the sorted values) in seconds (1 decimal)?
To do: What fraction of the requests took longer than 1 second?

Part 2. Do it in code

mean(values) and percentile(values, q): average, and q-th percentile with linear interpolation on the sorted values. ValueError for empty input.
fraction_over(values, limit): the fraction strictly above limit.
histogram(values, edges): counts of values in each bucket [edges[i], edges[i+1]); the last bucket also includes its upper edge. Values outside all buckets are counted in a final overflow entry, so the result has len(edges) entries.
percentile_from_histogram(counts, edges, q): the upper edge of the first bucket at which the cumulative count reaches q% of the total (an upper-bound estimate). Overflow returns float('inf').
ttft_and_gaps(request_time, token_times): (first token time − request time, list of gaps between neighbouring tokens).

def mean(values):
    pass

def percentile(values, q):
    pass

def fraction_over(values, limit):
    pass

def histogram(values, edges):
    pass

def percentile_from_histogram(counts, edges, q):
    pass

def ttft_and_gaps(request_time, token_times):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does mean latency alone establish a p95 target?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.