SD1.5 · LESSON 5 OF 6

Data and bandwidth

Turn a product requirement into a workload model, measurable objectives and a justified initial capacity estimate.

THE IDEA

Bits, bytes and the factor of eight

Two unit systems cause most capacity mistakes. Network links are rated in bits per second, files and disks in bytes, and a byte is 8 bits. A 1 Gbit/s link moves at most 125 MB/s. Separately, kilo, mega, giga and tera are powers of 1,000, while kibi, mebi, gibi and tebi are powers of 1,024, so 1 TiB is about 10% larger than 1 TB.

Value Unit

In Mbit/s100

In MB/s (decimal)12.5

In MiB/s (binary)11.92

Network links are rated in bits per second. Files and memory are counted in bytes, and one byte is 8 bits. Prefixes: kilo, mega, giga are powers of 1,000. Kibi, mebi, gibi, tebi are powers of 1,024. Try 100 Mbit/s, then 12.5 MB/s: the same traffic.

Convert any value between bit and byte units, decimal and binary. A frequent slip is reading MB/s as Mbit/s.

Predict first. A link is rated 1 Gbit/s. At best, roughly how many megabytes (decimal) can it carry per second?

What you store is bigger than what you receive

A 2.5 MB upload is never 2.5 MB on disk. Pixelfold also keeps thumbnails and resized copies (an assumed 30% on top of the original, so 3.25 MB) and each object is kept in three copies, so one photo takes 9.75 MB. At 100,000 uploads a day that is 975 GB a day and, over 365 days, 355.875 TB, which is 323.6 TiB.

Replicas 3Derivative factor 1.3x

original: 2.50 MBthumbnails and resized copies: 0.75 MBextra replicas: 6.50 MB

Payload the user sent2.50 MB

Bytes actually stored9.75 MB

One year, 100k uploads per day355.9 TB

323.7 TiB

Small records are worse: a like carries 16 bytes of information, yet with row and index overhead (assumed 200 bytes per copy) and 3 copies it takes 600 bytes, 37.5 times the payload.

Where the bytes of one photo go. Replication multiplies everything.

Metadata and overhead can dominate

For big blobs, metadata is negligible. For tiny records it is the whole story. A like carries two 8-byte ids, 16 bytes of information. A database row, its index entries and slack can be several times bigger. With an assumed 200 bytes per copy and 3 copies, a like costs 600 bytes: 37.5 times the payload. Protocol headers behave the same way on the network.

Chapter context

System design starts with what the system must do and the conditions under which it must do it. A component diagram is only useful when its choices can be traced back to access patterns, data volume, latency targets and failure expectations.

TRY IT YOURSELF

Assumptions for Pixelfold: an original photo is 2.5 MB (decimal), derivatives (thumbnails and resized copies) add 30% on top of the original, every object is kept in 3 copies, uploads are 100,000 per day and everything is retained for 365 days. A feed page shows 12 thumbnails of 120,000 bytes each. Opening one photo returns the 2.5 MB original. Peak rates for feed and photo opens come from the workload model: use the planning peak (average x 1.8 x 1.5).

Stored MB per photo One year, TB One year, TiB

Peak egress, Mbit/s Smallest link at 70% max Stored like vs payload, x

Value Unit

In Mbit/s100

In MB/s (decimal)12.5

In MiB/s (binary)11.92

Checks (0 of 6)

Bytes stored per uploaded photo, in decimal MB, after derivatives and 3 replicas
Photo storage after one year of retention, in decimal TB
The same amount in binary TiB (1 TiB = 2^40 bytes)
Peak origin egress in megabits per second (feed thumbnails plus full photo opens)
Choose the smallest link that keeps peak utilization at or below 70%
How many times larger is the stored like record (all replicas) than its 16-byte payload?

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does raw payload size include all database storage?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For U daily active users and a requests per user per day:

R_{avg}=Ua/86400,\quad R_{peak}=pR_{avg},\quad n=\lceil R_{peak}/(cu)\rceil.

Here p is a peak multiplier, c measured capacity per instance and u a chosen usable fraction. The multiplier and capacity are workload assumptions, not universal constants.

Worked example

Assume 100,000 daily users, 20 requests each and a peak multiplier of 10. Average demand is 2,000,000/86,400≈23.15 requests/s and estimated peak is 231.48 requests/s. An instance sustaining 100 requests/s with usable fraction 0.6 contributes 60 requests/s, so four instances cover the estimated peak. Five are needed to retain that same usable capacity after losing one instance.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil,isclose
average=100_000*20/86400
peak=10*average
usable=100*.6
assert isclose(average,23.14814814814815)
assert ceil(peak/usable) == 4
assert (5-1)*usable >= peak
assert (4-1)*usable < peak

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Average traffic is not peak traffic. A headroom assumption is not measured performance. Durability, correctness and availability need separate objectives.