E9.2 · LESSON 2 OF 6

Capacity and concurrency

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

Capacity, concurrency and queues

Required capacity follows from how fast requests arrive and how long each takes. Little's law says the average number in the system equals the arrival rate times the time each spends there: 4 per second × 2 seconds = 8 concurrent requests. But averages do not tell the whole story: a burst can exceed capacity even if the long-run average is fine, and as utilisation approaches 100% the waiting time grows very quickly.

arrivals per second4seconds in the system2

average number in the system = 4 × 2 = 8 (Little's law)

burst rate (per s) for 5 s, capacity 8/s12

backlog after the burst = (12 − 8) × 5 = 20 even though the long-run average (4/s) is below capacity

service rate μ (per s)10arrival rate λ9

utilisation 90% mean time in the system = 1/(μ − λ) = 1 s

Near full utilisation the waiting time grows without bound. Capacity must be planned for bursts and tails, not just averages.

Little's law, a burst backlog, and the queueing curve.

Predict first. Can average demand below a peak benchmark still produce overload bursts?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out concurrency, a burst backlog and queueing delay by hand, then simulate them.

arrivals per second4seconds in the system2

average number in the system = 4 × 2 = 8 (Little's law)

burst rate (per s) for 5 s, capacity 8/s12

backlog after the burst = (12 − 8) × 5 = 20 even though the long-run average (4/s) is below capacity

service rate μ (per s)10arrival rate λ9

utilisation 90% mean time in the system = 1/(μ − λ) = 1 s

Near full utilisation the waiting time grows without bound. Capacity must be planned for bursts and tails, not just averages.

4/s × 2 s backlog after 12/s for 5 s against 8/s seconds at μ 10, λ 9

To do: Requests arrive at 4 per second and each spends 2 seconds in the system. Average number of requests in the system (arrival rate × time)?
To do: Average demand is 4 per second, below the 8 per second the service can handle. But a burst brings 12 per second for 5 seconds. How many requests are waiting at the end of the burst?
To do: Queueing model (one server): mean time in system = 1 / (μ − λ). Service rate μ = 10 per second, arrival rate λ = 9. Seconds?

Part 2. Do it in code

littles_law(arrival_rate, seconds_in_system): average number in the system.
backlog_after(profile, capacity): profile is a list of (arrival_rate, seconds). Simulate second by second: the backlog grows by (arrival − capacity) per second and never goes below 0. Return the final backlog.
mm1_time(arrival_rate, service_rate): 1 / (service − arrival). ValueError if arrival >= service (the queue grows without bound).
utilization(arrival_rate, service_rate): arrival / service.
servers_needed(arrival_rate, service_rate_per_server, max_utilization=0.7): the smallest number of servers so that arrival / (servers · rate) <= max_utilization.

import math

def littles_law(arrival_rate, seconds_in_system):
    pass

def backlog_after(profile, capacity):
    pass

def mm1_time(arrival_rate, service_rate):
    pass

def utilization(arrival_rate, service_rate):
    pass

def servers_needed(arrival_rate, service_rate_per_server, max_utilization=0.7):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can average demand below a peak benchmark still produce overload bursts?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.