E9.1 · LESSON 1 OF 6

Cost units

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

What is a unit of cost?

"Cost per token" hides choices. Are input tokens counted, output tokens, or both? Which costs are in: the accelerator, the host, networking, staff? And what throughput do you divide by? Use the measured, sustained rate, not peak arithmetic capacity.

Worked example: $2/hour at 100 useful output tokens/s costs 2 × 1,000,000 / (3600 × 100) ≈ $5.5556 per million output tokens.

$ per hour$2sustained output tokens/s100input tokens per request800output tokens per request100peak tokens/s (marketing number)200

unit	formula	dollars
per million OUTPUT tokens	2 × 10⁶ / (3600 × 100)	5.5556
per request	1 requests/s	0.000556
per million COMBINED tokens	per request / (800 + 100) × 10⁶	0.6173
per million output tokens using the PEAK rate	optimistic	2.7778

State what is counted (input, output or both) and which costs are included, and use the sustained, measured rate. Peak arithmetic flatters the result.

The same machine under three definitions of a unit.

Predict first. Is dollars per million output tokens identical to dollars per million combined tokens?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute three cost units for the same machine by hand, then write the cost functions.

$ per hour$2sustained output tokens/s100input tokens per request800output tokens per request100peak tokens/s (marketing number)200

unit	formula	dollars
per million OUTPUT tokens	2 × 10⁶ / (3600 × 100)	5.5556
per request	1 requests/s	0.000556
per million COMBINED tokens	per request / (800 + 100) × 10⁶	0.6173
per million output tokens using the PEAK rate	optimistic	2.7778

State what is counted (input, output or both) and which costs are included, and use the sustained, measured rate. Peak arithmetic flatters the result.

$ per million output tokens ($2/h, 100/s) $ per request (1 request/s) $ per million combined tokens (900)

To do: A fictional machine costs $2 per hour and sustains 100 useful output tokens per second. Dollars per million OUTPUT tokens: 2 × 1,000,000 / (3600 × 100). (4 decimals)
To do: That machine serves one request per second, each producing 100 output tokens (and reading an 800-token prompt). Dollars per request (6 decimals)?
To do: Dollars per million COMBINED tokens (800 input + 100 output = 900 per request, 3 decimals)?

Part 2. Do it in code

cost_per_million(rate_per_hour, tokens_per_second): rate · 1,000,000 / (3600 · tokens_per_second). ValueError if the throughput is not positive.
cost_per_request(rate_per_hour, requests_per_second): rate / 3600 / requests_per_second.
cost_per_million_tokens(rate_per_hour, requests_per_second, tokens_per_request): the cost per million tokens when each request counts `tokens_per_request` tokens (use 100 for output only, 900 for input + output).
sustained_vs_peak(rate_per_hour, sustained_tps, peak_tps): a dict with cost_sustained and cost_peak; the peak figure is optimistic.

def cost_per_million(rate_per_hour, tokens_per_second):
    pass

def cost_per_request(rate_per_hour, requests_per_second):
    pass

def cost_per_million_tokens(rate_per_hour, requests_per_second, tokens_per_request):
    pass

def sustained_vs_peak(rate_per_hour, sustained_tps, peak_tps):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is dollars per million output tokens identical to dollars per million combined tokens?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.