Limits and isolation
Identify reusable prompt prefixes, calculate saved prefill work and distinguish token savings from hit rates.
THE IDEA
What caching does not do, and who may share
Prefix caching removes repeated prefill work. It does not shorten decoding: a cached prompt still has to generate every output token one by one. In the toy numbers, reusing 500 of 800 prompt tokens saves 50 ms of a 1480 ms request (3.4%); long outputs dominate.
Reuse also needs boundaries. Cache identity must include who may use an entry: one user's private prompt must not be served to another user just because the text matches, unless that content was deliberately made shareable. Reuse can also leave timing differences that reveal that a prompt was cached, which is another reason to scope the cache.
Predict first. Does a cached prompt make generating a long new response free?
Chapter context
Prefix caching reuses compatible KV state for a repeated beginning of a prompt. A repeated suffix alone is not enough, because causal hidden states depend on the preceding context.
TRY IT YOURSELF
Part 1. Do it hands-on
Cost a request with and without a hit, then write tenant-scoped cache keys.
- To do: Toy costs: 0.1 ms per prefill token, 14 ms per generated token. A request has an 800-token prompt and produces 100 tokens. Total time WITHOUT caching (ms)?
- To do: With 500 prompt tokens reused from the cache, total time (ms)?
- To do: Tenant A's private prompt blocks are cached. Tenant B sends an identical prompt. Should B be allowed to reuse A's private cache entries unless A's content was explicitly made shareable? Type yes or no
Part 2. Do it in code
- request_ms(prompt_tokens, reused_tokens, output_tokens, prefill_ms=0.1, decode_ms=14.0): (prompt − reused) · prefill_ms + output · decode_ms. ValueError if reused > prompt.
- saved_fraction(prompt_tokens, reused_tokens, output_tokens): how much of the uncached time the cache removes, as a fraction.
- cache_key(tenant, model, block_hash, shared=False): the key under which a block is stored. A private block's key includes the tenant; a block explicitly marked shared uses the tenant name "*" so all tenants can find it.
- can_reuse(owner_key, requester_tenant): True if owner_key was stored for the requester's tenant or for "*". The key is a tuple (tenant, model, hash).
- lookup(store, tenant, model, block_hash): return True if the requester can find the block (private key for them or the shared key).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does a cached prompt make generating a long new response free?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A simple saved-token estimate is
where H is the number of hits reusing P prefix tokens. If all R requests contain P prefix and U unique tokens, the baseline prompt work is and the nominal fraction saved is . This models token work, not directly wall-clock speed.
Worked example
One hundred requests each contain a 500-token common prefix and a 300-token unique suffix. Eighty hits reuse the entire prefix. Saved prefill tokens are 80×500=40,000 out of 100×800=80,000 baseline tokens: 50% token work saved despite an 80% request hit rate. Cache lookup and new-output generation still take work.
Reference code
The following example uses the Python standard library and does not call a remote service.
requests,hits,prefix,unique=100,80,500,300
saved=hits*prefix
baseline=requests*(prefix+unique)
assert saved == 40000 and baseline == 80000
assert saved/baseline == .5
assert hits/requests == .8
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Textual similarity is not cache identity. A common suffix is not a common causal prefix. Hit rates need definitions and do not directly translate into end-to-end speedups.