Special tokens and templates
Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.
THE IDEA
Tokens you never typed
A chat model does not receive your message alone. The conversation is wrapped in special tokens that mark who is speaking and where each message ends, and a final marker invites the model to answer. Those tokens are part of the model's contract: it was trained on one exact format, so the template must match.
The orange markers are a made-up toy template. Real models define their own role markers and special tokens, and you must use the one the model was trained with.
Predict first. Can formatting consume context even when the user text is short?
Chapter context
A language model receives token IDs, not visible words. The tokenizer chooses how text becomes those IDs. Different tokenizers can represent the same string with different lengths, so character count is not a universal token count.
TRY IT YOURSELF
Part 1. Do it hands-on
Count the tokens a toy template produces and find how many more turns fit. Then build the template and a trimmer in Python.
The orange markers are a made-up toy template. Real models define their own role markers and special tokens, and you must use the one the model was trained with.
- To do: Template: each message becomes <|role|> words… <|end|>, and a final <|assistant|> is appended. For system "Be brief." and user "Hi": how many tokens (each marker is one token, each word one token)?
- To do: The user typed only 1 token ("Hi") plus the system text's 2 words. How many tokens came from the template itself?
- To do: The context budget is 20 tokens. Every further user turn "Hi" costs <|user|> Hi <|end|> = 3 tokens. How many more turns fit after the first conversation?
Part 2. Do it in code
- chat_tokens(messages): messages is a list of dicts with role and content. For each message output "<|role|>", then each whitespace-separated word of the content, then "<|end|>". After all messages append "<|assistant|>". Return the list of tokens.
- overhead(messages): how many tokens in chat_tokens are not words from the messages' content (the markers).
- turns_that_fit(messages, budget, user_text): how many additional user messages with user_text can be appended before the total would exceed the budget. Return 0 if the conversation itself does not fit.
- trim_to_budget(messages, budget): drop the OLDEST non-system messages until chat_tokens fits the budget. System messages are always kept. Raise ValueError if it still does not fit. Return a new list.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Can formatting consume context even when user text is short?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
In a toy merge, replacing m nonoverlapping adjacent pairs by one symbol reduces sequence length by m. A request budget must include formatted input plus requested output within the model's supported context; exact accounting is model-specific.
Worked example
Start with toy symbols ['a','b','a','b'] and a vocabulary containing a and b. The pair (a,b) occurs twice. A merge creates symbol ab and encodes the sequence as ['ab','ab'], reducing length from 4 to 2. This deliberately omits byte encoding and pre-tokenization so the merge operation is visible.
Reference code
The following example uses the Python standard library and does not call a remote service.
tokens = list('abab')
merged=[]
i=0
while i < len(tokens):
if tokens[i:i+2] == ['a','b']:
merged.append('ab'); i += 2
else:
merged.append(tokens[i]); i += 1
assert merged == ['ab','ab']
assert len(tokens)-len(merged) == 2
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A token is not necessarily a word. A vocabulary ID is a label, not a magnitude. A hand-built toy merge sequence is not a drop-in replacement for a production tokenizer.