E8.6 · LESSON 6 OF 6

Engine comparisons

Compare kernel, parallelism and disaggregated-serving choices using memory, communication and latency constraints.

THE IDEA

Comparing serving engines fairly

Engines differ in kernels, scheduling, cache policy and supported features, so benchmark numbers are easy to misread. A fair comparison uses the same model at equivalent output quality, the same input and output lengths, the same concurrency and hardware, discards warm-up, states the cache settings, and judges every engine against the same latency objective.

p95 objective (s)1

engine	tokens/s	p95 latency (s)	meets the objective?
A	1000	0.8	yes
B	1200	1.5	no

best among those meeting p95 ≤ 1 s: engine A

A fair comparison fixes the model and its quality, input and output lengths, concurrency, hardware, warm-up and cache policy. Two unlike traces cannot justify a direct ranking. Drop warm-up samples: for the samples 5, 5, 1, 1, 1, 1, 1, 1, 1, 1 the mean is 1.8 with warm-up and 1.0 without.

Engine B is faster in raw throughput. Move the objective.

Predict first. Can two unlike workload traces justify a direct engine ranking?

Chapter context

Scaling serving changes how work is placed and moved. An operation can be mathematically unchanged while its memory traffic changes dramatically. Device count alone does not describe the performance of the resulting system.

TRY IT YOURSELF

Part 1. Do it hands-on

Judge two engines under an objective by hand, then write the comparability and ranking checks.

p95 objective (s)1

engine	tokens/s	p95 latency (s)	meets the objective?
A	1000	0.8	yes
B	1200	1.5	no

best among those meeting p95 ≤ 1 s: engine A

engines meeting p95 ≤ 1.0 faster among those (A or B) mean including warm-up

To do: Engine A: 1000 tokens/s with p95 latency 0.8 s. Engine B: 1200 tokens/s with p95 1.5 s. The objective is p95 ≤ 1.0 s. How many of the two engines meet it?
To do: Which engine has the higher throughput among those that meet the objective? Type the letter
To do: Latency samples in order: 5, 5, 1, 1, 1, 1, 1, 1, 1, 1 (s). The first two are warm-up. Mean INCLUDING warm-up?

Part 2. Do it in code

comparable(a, b): a and b are dicts describing a benchmark run. They are comparable only if they have the same model, quality (the same measured quality, e.g. an accuracy), input_len, output_len, concurrency and hardware, and both have all of those fields (None counts as missing).
trim_warmup(samples, n): drop the first n samples. ValueError if n is not smaller than the number of samples.
mean(values) and p95(values): the average and the 95th percentile (linear interpolation on the sorted values).
best_under_slo(runs, slo_p95): among the runs (dicts with name, tps, p95) whose p95 is at most the objective, return the name of the one with the highest tps (the first on ties); None if none qualifies.
ranking_justified(run_a, run_b): True only if the two runs are comparable AND both specify a warmup count.

def comparable(a, b):
    pass

def trim_warmup(samples, n):
    pass

def mean(values):
    pass

def p95(values):
    pass

def best_under_slo(runs, slo_p95):
    pass

def ranking_justified(run_a, run_b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can two unlike workload traces justify a direct engine ranking?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A weight-traffic lower-bound estimate is

t_{step}\gtrsim M_{read}/BW.

This assumes the stated bytes really traverse the bottleneck bandwidth. KV transfer adds roughly bytes_to_move/network_bandwidth before protocol and synchronization overhead. Batching and caching can change the bytes per produced token.

Worked example

A fictional 14-GB dense weight scan across a sustained 700-GB/s memory path takes at least 14/700=0.02 seconds, giving a 50-step/s bandwidth ceiling under the one-scan-per-step assumption. Real execution includes other memory traffic and work. A 512-MiB KV transfer over 32 GiB/s has a payload-only lower bound of 1/64 second.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
assert isclose(14/700,.02)
assert isclose(1/(14/700),50)
transfer=(512*1024**2)/(32*1024**3)
assert isclose(transfer,1/64)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Peak device bandwidth is not guaranteed sustained application bandwidth. Exact attention can be computed with different memory layouts. Additional GPUs can add communication bottlenecks.