E5.5 · LESSON 5 OF 6

Prompt layout

Identify reusable prompt prefixes, calculate saved prefill work and distinguish token savings from hit rates.

THE IDEA

Prompt layout and caching

Because only the beginning can be reused, the order of the parts of a prompt decides how much is shared. A per-request identifier, timestamp or random value at the front makes every prompt differ from the first token, so nothing behind it can be reused. Where the task allows it, put the stable shared material first and the variable material later.

request id

system prompt

shared document

question

order: request id (5) → system prompt (300) → shared document (200) → question (20) tokens identical across two different requests at the start: 0 (green = same for every request, orange = differs per request)

One variable item at the very front breaks sharing for everything behind it. Put stable material first when the task allows it, but never reorder instructions in a way that changes how the model reads them just to get a hit.

The same four parts in two orders.

Predict first. Can putting a random identifier first defeat later shared-prefix reuse?

Chapter context

Prefix caching reuses compatible KV state for a repeated beginning of a prompt. A repeated suffix alone is not enough, because causal hidden states depend on the preceding context.

TRY IT YOURSELF

Part 1. Do it hands-on

Measure the shared start for two layouts, then write the layout helpers.

request id

system prompt

shared document

question

shared start, id first shared start, id moved reorder the instructions just for a hit? (yes/no)

To do: Each prompt is: random request id (5 tokens), system prompt (300 tokens), shared document (200 tokens), question (20 tokens). Two different requests: how many leading tokens are identical?
To do: Move the request id after the shared document, keeping the same meaning. Now how many leading tokens are identical between two requests?
To do: Is it acceptable to reorder the instructions inside the system prompt just to make a longer shared prefix, if that changes how the model reads them? Type yes or no

Part 2. Do it in code

build_prompt(layout, values): layout is a list of part names; values maps a part name to its list of tokens. Return the concatenated token list.
shared_prefix_tokens(layout, req_a, req_b): the length of the common prefix of the prompts built for two requests (each a values dict).
move_variable_last(parts): parts is a list of dicts with name, stable (True if the content is identical across requests) and movable (True if moving it does not change the task's meaning). Return the names in a new order: every part keeps its place EXCEPT movable, non-stable parts, which are moved after the last stable part. Non-movable parts never move.
reuse_gain(layout_before, layout_after, req_a, req_b): shared_prefix_tokens(after) − shared_prefix_tokens(before).

def build_prompt(layout, values):
    pass

def shared_prefix_tokens(layout, req_a, req_b):
    pass

def move_variable_last(parts):
    pass

def reuse_gain(layout_before, layout_after, req_a, req_b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can putting a random identifier first defeat later shared-prefix reuse?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple saved-token estimate is

S=H\,P,

where H is the number of hits reusing P prefix tokens. If all R requests contain P prefix and U unique tokens, the baseline prompt work is $R(P+U)$ and the nominal fraction saved is $HP/[R(P+U)]$ . This models token work, not directly wall-clock speed.

Worked example

One hundred requests each contain a 500-token common prefix and a 300-token unique suffix. Eighty hits reuse the entire prefix. Saved prefill tokens are 80×500=40,000 out of 100×800=80,000 baseline tokens: 50% token work saved despite an 80% request hit rate. Cache lookup and new-output generation still take work.

Reference code

The following example uses the Python standard library and does not call a remote service.

requests,hits,prefix,unique=100,80,500,300
saved=hits*prefix
baseline=requests*(prefix+unique)
assert saved == 40000 and baseline == 80000
assert saved/baseline == .5
assert hits/requests == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Textual similarity is not cache identity. A common suffix is not a common causal prefix. Hit rates need definitions and do not directly translate into end-to-end speedups.