E9.3 · LESSON 3 OF 6

Utilization

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

Utilisation and the real price of a result

If a machine is billed for the whole hour, every idle moment is paid for. The real cost per token is the bill divided by the tokens actually produced. If billed time stays fixed and useful throughput halves, cost per token doubles ($5.56 to $11.11 in the example). Realised throughput depends on the traffic: cache hits, output lengths and batch shapes all move it.

share of billed time spent serving100%tokens/s while serving100

billed for the hour

actually serving

useful rate over billed time = 100 × 1 = 100 tokens/s cost = $2/h → $5.5556 per million useful output tokens (baseline at 100% busy and 100 tokens/s: $5.5556)

Idle capacity that is billed still costs money. Cache hits, output lengths and batch shapes all change the realised throughput, and so the cost of a result.

Lower the share of time the machine is serving and watch the unit cost climb.

Predict first. What happens to cost per token if billed time stays fixed and useful throughput halves?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute costs with idle time and fewer useful tokens, then code the effective-cost functions including a cache effect.

share of billed time spent serving100%tokens/s while serving100

billed for the hour

actually serving

useful rate over billed time = 100 × 1 = 100 tokens/s cost = $2/h → $5.5556 per million useful output tokens (baseline at 100% busy and 100 tokens/s: $5.5556)

Idle capacity that is billed still costs money. Cache hits, output lengths and batch shapes all change the realised throughput, and so the cost of a result.

$ per million at 50 tokens/s $ per million when busy 25% billed hours for 10⁶ tokens at 100/s

To do: Billed time is fixed at $2/hour but useful throughput falls from 100 to 50 tokens/s. Dollars per million output tokens (4 decimals)?
To do: The machine is billed all hour, but is busy serving only 25% of the time (100 tokens/s while busy). Effective useful rate = 25 tokens/s. Dollars per million output tokens (4 decimals)?
To do: Hours of billed time needed to produce one million output tokens at a sustained 100 tokens/s (3 decimals)?

Part 2. Do it in code

cost_per_million(rate_per_hour, tps) as before.
effective_tps(tps_when_busy, busy_fraction): useful tokens per second averaged over billed time. busy_fraction must be in (0, 1].
effective_cost(rate_per_hour, tps_when_busy, busy_fraction): cost per million useful tokens when the machine is billed all the time.
billed_hours(tokens, tps): hours of billed time to produce `tokens` at a sustained rate.
cost_with_cache(rate_per_hour, requests_per_second, prefill_tokens, output_tokens, hit_rate, reused_tokens, prefill_ms=0.1, decode_ms=14.0): cost per request when a share `hit_rate` of requests reuses `reused_tokens` prefix tokens; machine seconds per request = ((prefill − hit_rate·reused)·prefill_ms + output·decode_ms)/1000, cost = seconds · rate/3600.

def cost_per_million(rate_per_hour, tps):
    pass

def effective_tps(tps_when_busy, busy_fraction):
    pass

def effective_cost(rate_per_hour, tps_when_busy, busy_fraction):
    pass

def billed_hours(tokens, tps):
    pass

def cost_with_cache(rate_per_hour, requests_per_second, prefill_tokens, output_tokens, hit_rate, reused_tokens, prefill_ms=0.1, decode_ms=14.0):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What happens to cost per token if billed time stays fixed and useful throughput halves?

AIt stays the same, because the bill is fixed.BIt doubles, because the same bill is divided by half as many tokens.CIt halves, because fewer tokens means less cost.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.