E5.4 · LESSON 4 OF 6

Hit rates and saved tokens

Identify reusable prompt prefixes, calculate saved prefill work and distinguish token savings from hit rates.

THE IDEA

Hit rate is not the same as work saved

A request hit rate counts requests that found a cached prefix. Saved prefill tokens count how much work that actually removed, which depends on how long the reused prefix was compared with the whole prompt. In the chapter's example 80 of 100 requests hit and reuse 500 tokens, but each prompt has 800 tokens: 40,000 of 80,000 prefill tokens are saved, which is 50%, not 80%.

requests100request hit rate80%reused prefix tokens500unique suffix tokens300

request hit rate

80%

prefill tokens saved

50%

80 hits × 500 = 40,000 tokens saved of 100 × 800 = 80,000 baseline → 50% A high hit rate saves little when the shared prefix is short compared with the unique part.

Drag the hit rate and the prefix length, and compare the two bars.

Predict first. Is an 80% request hit rate necessarily an 80% token-work saving?

Chapter context

Prefix caching reuses compatible KV state for a repeated beginning of a prompt. A repeated suffix alone is not enough, because causal hidden states depend on the preceding context.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute the chapter's saving by hand, then write the hit-rate and saved-token functions.

requests100request hit rate80%reused prefix tokens500unique suffix tokens300

request hit rate

80%

prefill tokens saved

50%

80 hits × 500 = 40,000 tokens saved of 100 × 800 = 80,000 baseline → 50% A high hit rate saves little when the shared prefix is short compared with the unique part.

tokens saved (80 hits × 500) baseline tokens (100 × 800) saving in percent

To do: 100 requests each have a 500-token common prefix and a 300-token unique suffix. 80 of them hit the cache and reuse the whole prefix. Prefill tokens saved?
To do: Prefill tokens without any caching?
To do: Share of prefill token work saved, in percent (while the REQUEST hit rate is 80%)?

Part 2. Do it in code

request_hit_rate(hits, total): hits / total.
saved_tokens(reused_per_request): the sum of the reused prefix lengths over all requests (0 for misses).
baseline_tokens(prompt_lengths): the sum of all prompt lengths (no caching).
token_saving_fraction(reused_per_request, prompt_lengths): saved / baseline.
equal_prefix_workload(total, hits, prefix, suffix): build both lists for the chapter's workload and return (hit_rate, token_saving_fraction).

def request_hit_rate(hits, total):
    pass

def saved_tokens(reused_per_request):
    pass

def baseline_tokens(prompt_lengths):
    pass

def token_saving_fraction(reused_per_request, prompt_lengths):
    pass

def equal_prefix_workload(total, hits, prefix, suffix):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is an 80% request hit rate necessarily an 80% token-work saving?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple saved-token estimate is

S=H\,P,

where H is the number of hits reusing P prefix tokens. If all R requests contain P prefix and U unique tokens, the baseline prompt work is $R(P+U)$ and the nominal fraction saved is $HP/[R(P+U)]$ . This models token work, not directly wall-clock speed.

Worked example

One hundred requests each contain a 500-token common prefix and a 300-token unique suffix. Eighty hits reuse the entire prefix. Saved prefill tokens are 80×500=40,000 out of 100×800=80,000 baseline tokens: 50% token work saved despite an 80% request hit rate. Cache lookup and new-output generation still take work.

Reference code

The following example uses the Python standard library and does not call a remote service.

requests,hits,prefix,unique=100,80,500,300
saved=hits*prefix
baseline=requests*(prefix+unique)
assert saved == 40000 and baseline == 80000
assert saved/baseline == .5
assert hits/requests == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Textual similarity is not cache identity. A common suffix is not a common causal prefix. Hit rates need definitions and do not directly translate into end-to-end speedups.