Data parallel replicas
Compare kernel, parallelism and disaggregated-serving choices using memory, communication and latency constraints.
THE IDEA
Data-parallel replicas
The simplest way to serve more traffic is to run several complete copies of the model, replicas, each handling its own requests. Aggregate capacity grows with the number of replicas (three replicas of 70 tokens/s give 210 tokens/s). But a single request still runs on one replica and passes through every layer, so its own speed does not improve.
32 layers
32 layers
32 layers
Replicas serve separate requests with their own copy of the model. They raise how much work the service can take at once; they do not make one request's forward path shorter.
Predict first. Does a second identical replica halve one request's layer count?
Chapter context
Scaling serving changes how work is placed and moved. An operation can be mathematically unchanged while its memory traffic changes dramatically. Device count alone does not describe the performance of the resulting system.
TRY IT YOURSELF
Part 1. Do it hands-on
Compute capacity and latency with replicas, then code the capacity-planning helpers.
32 layers
32 layers
32 layers
Replicas serve separate requests with their own copy of the model. They raise how much work the service can take at once; they do not make one request's forward path shorter.
- To do: Each replica generates 70 tokens/s in total across its requests. With 3 identical replicas and enough requests, aggregate tokens/s?
- To do: One request generates 100 tokens on ONE replica at 70 tokens/s for that request. Seconds with 3 replicas (a replica serves it alone)? (2 decimals)
- To do: A model has 32 layers. A second identical replica is added. How many layers does a request still pass through?
Part 2. Do it in code
- aggregate_tps(replicas, per_replica_tps): replicas · per_replica_tps (assuming enough requests to keep them all busy).
- request_seconds(output_tokens, tps_per_request): output_tokens / tps_per_request. (Adding replicas does not change it.)
- replicas_needed(target_tps, per_replica_tps): ceil(target / per replica).
- layers_per_request(layers, replicas): the layers one request passes through: always `layers` (a replica is a full copy).
- round_robin(n_requests, replicas): the number of requests each replica receives when they are dealt out in turn; a list of length `replicas`.
- imbalance(counts): max(counts) / mean(counts).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does a second identical replica halve one request's layer count?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A weight-traffic lower-bound estimate is
This assumes the stated bytes really traverse the bottleneck bandwidth. KV transfer adds roughly bytes_to_move/network_bandwidth before protocol and synchronization overhead. Batching and caching can change the bytes per produced token.
Worked example
A fictional 14-GB dense weight scan across a sustained 700-GB/s memory path takes at least 14/700=0.02 seconds, giving a 50-step/s bandwidth ceiling under the one-scan-per-step assumption. Real execution includes other memory traffic and work. A 512-MiB KV transfer over 32 GiB/s has a payload-only lower bound of 1/64 second.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import isclose
assert isclose(14/700,.02)
assert isclose(1/(14/700),50)
transfer=(512*1024**2)/(32*1024**3)
assert isclose(transfer,1/64)
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Peak device bandwidth is not guaranteed sustained application bandwidth. Exact attention can be computed with different memory layouts. Additional GPUs can add communication bottlenecks.