Quality requirements
Turn a product requirement into a workload model, measurable objectives and a justified initial capacity estimate.
THE IDEA
Four promises, four numbers
"High quality" breaks into properties that fail independently. Latency is how long an operation takes, usually stated as a percentile because the slowest 1% matters. Availability is the share of time or requests that succeed. Durability is whether data that was acknowledged still exists later. Correctness (here: freshness) is whether the answer is right. A service can be fast and wrong, up and lossy, or correct and slow.
A service level objective (SLO) is a target for a measured indicator, such as "99% of feed loads under 400 ms over a month". Targets are only useful if each can be measured and if breaching it matters to users.
One incident, four dashboards
Select an incident. All four properties are shown against one example set of objectives. Notice which dashboard goes red, and that it is not always the same one.
A cache setting is wrong. Feeds come back fast, but they are up to 10 minutes old.
Latency is better than normal and the service is up, yet the data is wrong. Only the freshness objective sees it.
Predict first. A deploy makes the feed come back in 180 ms, but with data 10 minutes old. Does a latency objective of 500 ms notice?
Targets cost money: the error budget
An availability target implies how much failure you may spend. 99.9% over a 30-day month (43,200 minutes) allows 43.2 minutes of downtime. 99.99% allows 4.32 minutes. Each extra nine divides the budget by ten and typically needs redundancy, automation and faster response.
Every extra nine divides the budget by ten, and usually multiplies the cost. Choose the target users actually need.
Chapter context
System design starts with what the system must do and the conditions under which it must do it. A component diagram is only useful when its choices can be traced back to access patterns, data volume, latency targets and failure expectations.
TRY IT YOURSELF
Pixelfold has four separate promises to keep. Set an objective for each. The replay below applies your objectives to a normal month and to four incidents. A good set stays quiet on a normal month and alarms on every incident that hurt users.
| Replay | p99 | avail. | feed age | lost | Your SLOs |
|---|---|---|---|---|---|
| Normal month (baseline) | 420 ms | 99.950% | 20 s | 0 | quiet (good) |
| I1: Stale feed after a cache change | 180 ms | 99.950% | 600 s | 0 | not caught |
| I2: Disk failure with lagging replica | 400 ms | 99.950% | 20 s | 40 | not caught |
| I3: Slow database queries | 900 ms | 99.950% | 20 s | 0 | not caught |
| I4: Bad deploy, full outage | 420 ms | 99.884% | 20 s | 0 | not caught |
Checks (0 of 5)
- Latency SLO: a normal day passes and the slow-database incident (I3) is caught
- Availability SLO: a normal month passes and the 50-minute outage (I4) is caught
- Freshness SLO: a normal day passes and the stale-feed incident (I1) is caught
- Durability: the lost-uploads incident (I2) is caught
- Enter the monthly error budget in minutes for your availability target
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Is availability identical to durability?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
For U daily active users and a requests per user per day:
Here p is a peak multiplier, c measured capacity per instance and u a chosen usable fraction. The multiplier and capacity are workload assumptions, not universal constants.
Worked example
Assume 100,000 daily users, 20 requests each and a peak multiplier of 10. Average demand is 2,000,000/86,400≈23.15 requests/s and estimated peak is 231.48 requests/s. An instance sustaining 100 requests/s with usable fraction 0.6 contributes 60 requests/s, so four instances cover the estimated peak. Five are needed to retain that same usable capacity after losing one instance.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import ceil,isclose
average=100_000*20/86400
peak=10*average
usable=100*.6
assert isclose(average,23.14814814814815)
assert ceil(peak/usable) == 4
assert (5-1)*usable >= peak
assert (4-1)*usable < peak
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Average traffic is not peak traffic. A headroom assumption is not measured performance. Durability, correctness and availability need separate objectives.