SD1.4 · LESSON 4 OF 6

Capacity and headroom

Turn a product requirement into a workload model, measurable objectives and a justified initial capacity estimate.

THE IDEA

Why you cannot run servers at 100%

A single instance with a measured service rate of 100 requests per second does not give good latency at 100 requests per second. As the load approaches capacity, requests start to queue and latency bends sharply upward. Below, the same instance is modelled with a simple queueing approximation. Drag utilization up and watch where the 50 ms target is lost.

Utilization of one instance 60%

one instance, 100 req/s service rate

Mean latency here25.0 ms

Meets 50 ms target?yes

Model: mean latency = (1/rate) / (1 - utilization), with a 10 ms service time. It is a simplified queueing approximation, not a benchmark. Real services have their own curve, which you must measure. The shape is the lesson: latency is flat, then bends sharply.

Latency is flat, then it climbs. At 80% utilization the mean is exactly 50 ms in this model.

That is why a capacity plan uses a usable fraction of the measured capacity, the operating point you are willing to run at. It is a decision about risk, not a measurement.

Sizing, step by step

Divide the planning peak by usable capacity per instance, round up, then add a spare so that losing one instance does not break the objective.

Usable fraction of measured capacity 0.80

Planning peak from the workload model: 246.9 req/s.
One instance is measured at 100 req/s. At 0.80 usable it contributes 80.0 req/s.
Instances for the peak: ceil(246.9 / 80.0) = 4. Without a spare, lose one and capacity drops to 240.0 req/s, which is below the peak.
With one spare (N+1): 5 instances. After losing one, 320.0 req/s remain, which covers the peak.

A lower usable fraction buys more headroom for variability and costs more instances. The fraction is a decision, and the 100 req/s is an assumed benchmark.

Change the usable fraction and see how many instances the arithmetic asks for.

Predict first. Four instances cover the planning peak exactly as the arithmetic says (at 80% usable). One of them fails during the busiest hour. What happens?

An estimate is not a benchmark

The 100 requests per second came from an assumed benchmark. In real work, you measure the per-instance capacity of your own code, with your own request mix, at the latency you promise, and you re-measure after big changes. Treat the estimate as a starting point to be checked by a load test.

Chapter context

System design starts with what the system must do and the conditions under which it must do it. A component diagram is only useful when its choices can be traced back to access patterns, data volume, latency targets and failure expectations.

TRY IT YOURSELF

The planning peak is about 246.9 requests per second. One instance was benchmarked at 100 req/s, and the latency target is 50 ms mean (utilization of at most 80%). The design below has too few instances. Size it, then prove it by running a whole day, and again with a failure.

Instances 2N+1 count at 80% usable

No run yet. Press a run button to see the day.

Checks (0 of 4)

Calculate the instances needed with one spare (N+1) at 80% usable capacity
Run a normal day with your current instance count: no hour breaks the 50 ms target
Run the day with one instance lost during hours 19 to 21: still no hour breaks the target
Meet both runs with at most 6 instances (the budget cap)

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should a capacity plan assume every server runs at its failure limit?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For U daily active users and a requests per user per day:

R_{avg}=Ua/86400,\quad R_{peak}=pR_{avg},\quad n=\lceil R_{peak}/(cu)\rceil.

Here p is a peak multiplier, c measured capacity per instance and u a chosen usable fraction. The multiplier and capacity are workload assumptions, not universal constants.

Worked example

Assume 100,000 daily users, 20 requests each and a peak multiplier of 10. Average demand is 2,000,000/86,400≈23.15 requests/s and estimated peak is 231.48 requests/s. An instance sustaining 100 requests/s with usable fraction 0.6 contributes 60 requests/s, so four instances cover the estimated peak. Five are needed to retain that same usable capacity after losing one instance.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil,isclose
average=100_000*20/86400
peak=10*average
usable=100*.6
assert isclose(average,23.14814814814815)
assert ceil(peak/usable) == 4
assert (5-1)*usable >= peak
assert (4-1)*usable < peak

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Average traffic is not peak traffic. A headroom assumption is not measured performance. Durability, correctness and availability need separate objectives.