Static batches
Trace static and iteration-level schedules and explain chunked prefill, preemption and latency tradeoffs.
THE IDEA
Static batches wait for the slowest
The simplest scheduler forms a batch of requests, runs it until every request in it is finished, then forms the next. All requests in a batch advance one token per iteration, but they do not need the same number of tokens. When a short request finishes, its slot sits idle until the longest request is done.
| it 1 | it 2 | it 3 | it 4 | |
|---|---|---|---|---|
| slot 1 | A | A | C | C |
| slot 2 | B |
Grey cells are idle: a short request finishes but its slot cannot be given to a waiting request until the whole group is done.
Predict first. Why does one short request not always free capacity for a waiting request?
Chapter context
Requests arrive at different times and finish after different output lengths. Scheduling decides which work enters each iteration and how waiting prompts compete with ongoing generation.
TRY IT YOURSELF
Part 1. Do it hands-on
Count iterations and utilisation for the chapter's example by hand, then simulate a static scheduler.
| it 1 | it 2 | it 3 | it 4 | |
|---|---|---|---|---|
| slot 1 | A | A | C | C |
| slot 2 | B |
Grey cells are idle: a short request finishes but its slot cannot be given to a waiting request until the whole group is done.
- To do: Two slots serve requests A, B, C needing 2, 1 and 2 iterations. A static schedule runs A+B together until BOTH are done (2 iterations), then C alone until done (2 iterations). Total iterations?
- To do: Slot-iterations that actually do work (2 + 1 + 2 = 5 token steps)? And the capacity is 4 iterations × 2 slots = 8. Type the number of ACTIVE slot-iterations
- To do: Utilisation = active / capacity, in percent
Part 2. Do it in code
- simulate_static(lengths, slots): requests are taken in order in groups of `slots`. A group runs until its LONGEST request is done, then the next group starts. Return a dict: iterations (total), active (slot-iterations in which a request really generates), capacity (iterations · slots), utilization (active / capacity) and finish (a list: the iteration at which each request finishes, 1-based, i.e. start of its group + its own length). Raise ValueError if slots < 1 or a length < 1.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Why does one short request not always free capacity for a waiting request?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
In a simplified equal-cost decode schedule, utilization = active sequence-iterations / available sequence-iterations. This omits prefill costs and assumes uniform iteration time. A real scheduler needs measured compute, memory and latency constraints.
Worked example
Two slots serve requests A,B,C with output lengths 2,1,2. Static groups run A+B for two iterations, then C alone for two: 5 active slots out of 8 available. Continuous admission runs A+B, then A+C, then C: 5 active slots out of 6. Utilization rises from 62.5% to about 83.33% in this toy schedule; real token costs are not equal.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import isclose
static=[['A','B'],['A',None],['C',None],['C',None]]
continuous=[['A','B'],['A','C'],['C',None]]
util=lambda rows: sum(x is not None for row in rows for x in row)/(2*len(rows))
assert util(static) == .625
assert isclose(util(continuous),5/6)
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Continuous batching does not guarantee a fixed improvement. Chunked prefill changes scheduling granularity, not the model's context dependencies. Include cache movement or recomputation costs when comparing preemption policies.