E7.6 · LESSON 6 OF 6

Variants and measurement

Trace draft-and-verify decoding, explain distribution-preserving acceptance, and calculate a simplified speedup.

THE IDEA

Variants and honest measurement

Besides a separate small draft model there are self-speculation (the target drafts for itself, for example with extra prediction heads or by skipping layers) and tree proposals (several candidate continuations are checked together). They keep different extra state and have different verification costs, so they do not share one draft-cost curve.

implementation	acceptance a	ms per draft step	tokens/s (k = 4)
A: separate small model	0.8	1	210.1
B: very cheap draft (e.g. reusing the target's own states)	0.6	0.2	180.1

one number is not enough: A has the higher acceptance rate AND the higher throughput here, but with other costs the order can flip. Variants differ in what extra state they keep and what verification costs: a separate draft model, self-speculation and tree proposals (several candidates checked at once) do not share one cost curve.

Compare variants on measured workloads: tokens per second, acceptance rate, memory used, and (for sampling methods) a check that the output distribution still matches the target.

Two implementations of the same idea with different costs.

Predict first. Does every speculative implementation have the same draft-cost curve?

Chapter context

A cheap draft proposes several tokens and a target model verifies them with parallel work. The opportunity comes from amortizing an expensive target step, but drafting, verification and rejection recovery all cost time.

TRY IT YOURSELF

Part 1. Do it hands-on

Compare two implementations by throughput, then write a log summariser that shows when the headline acceptance rate misleads.

implementation	acceptance a	ms per draft step	tokens/s (k = 4)
A: separate small model	0.8	1	210.1
B: very cheap draft (e.g. reusing the target's own states)	0.6	0.2	180.1

Compare variants on measured workloads: tokens per second, acceptance rate, memory used, and (for sampling methods) a check that the output distribution still matches the target.

B tokens/s (1 decimal) faster, A or B does the draft choice change the exact method's output distribution? (yes/no)

To do: Implementation B drafts much more cheaply (0.2 ms per step) but is accepted less often (a = 0.6). With k = 4 and 12 ms verification, tokens per second (1 decimal)?
To do: Implementation A (a = 0.8, 1 ms per step) makes 210.1 tokens/s. Which is faster on this workload, A or B? Type the letter
To do: In exact speculative sampling, does the choice of draft model change the DISTRIBUTION of the final text (only the speed)? Type yes or no (yes = the distribution changes)

Part 2. Do it in code

summarize(rounds): rounds is a list of dicts with k (drafted), accepted and ms (the time of that round). Return a dict with tokens (Σ accepted + number of rounds: each round also yields one target token), total_ms, tokens_per_second, acceptance_rate (Σ accepted / Σ k) and tokens_per_round. ValueError for an empty list.
compare(a_rounds, b_rounds): return "A" or "B" for the higher tokens_per_second ("tie" if equal within 1e-9).
same_distribution(freq_a, freq_b, tol): True if the two frequency vectors differ by at most tol in every entry (used to check that two decoders sample the same text distribution).
headline_misleads(rounds_a, rounds_b): True if the implementation with the HIGHER acceptance rate has the LOWER tokens per second.

def summarize(rounds):
    pass

def compare(a_rounds, b_rounds):
    pass

def same_distribution(freq_a, freq_b, tol):
    pass

def headline_misleads(rounds_a, rounds_b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does every implementation have the same draft-cost curve?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Under constant independent acceptance a and k draft proposals:

E[N]=\sum_{i=0}^{k}a^i=\frac{1-a^{k+1}}{1-a}\quad(a\ne1).

At a=1 the sum is k+1. A simple speed estimate is $E[N]/(t_{verify}+k t_{draft})$ . Exact samplers use conditional probabilities; this constant-a model is not their full correctness proof.

Worked example

For a=0.8,k=4, expected produced tokens are 1+0.8+0.64+0.512+0.4096=3.3616. If target-only generation takes 10 ms/token, verification costs 12 ms and four draft steps cost 1 ms each, speculative throughput is 3.3616/0.016=210.1 tokens/s versus 100 tokens/s baseline: an illustrative 2.101× speedup.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
a,k=.8,4
expected=sum(a**i for i in range(k+1))
assert isclose(expected,3.3616)
assert isclose(expected/.016,210.1)
assert isclose((expected/.016)/(1/.010),2.101)
assert sum(1**i for i in range(k+1)) == 5

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Expected accepted draft tokens alone omit the final target token in this model. a=1 needs the finite-sum limit. A correctness-preserving acceptance rule is more than an agreement heuristic.