Token budgets and limitations
Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.
THE IDEA
Budgets are in tokens, not words
Limits and prices are counted in tokens. A common word may be a single token while a rare word is several, so the same number of words can cost very different amounts. Tokenization also shapes what the model can directly see: it reads pieces, not letters.
| word | pieces the model sees | tokens | fits? |
|---|---|---|---|
| the | the | 1 | yes |
| strawberry | strawberry | 3 | yes |
| tokenization | tokenization | 2 | cut |
| is | is | 1 | cut |
| fun | fun | 1 | cut |
"strawberry" has 3 letter r's, but the model reads str | aw | berry; two pieces contain an r and none says how many. Tokenization is one reason letter counting is hard for models. It is not the only one.
Predict first. Does a token budget guarantee a fixed number of visible words?
Chapter context
A language model receives token IDs, not visible words. The tokenizer chooses how text becomes those IDs. Different tokenizers can represent the same string with different lengths, so character count is not a universal token count.
TRY IT YOURSELF
Part 1. Do it hands-on
Count tokens with the toy table, work out what fits in a budget, and see what the model can and cannot see in a word. Then code it.
| word | pieces the model sees | tokens | fits? |
|---|---|---|---|
| the | the | 1 | yes |
| strawberry | strawberry | 3 | yes |
| tokenization | tokenization | 2 | cut |
| is | is | 1 | cut |
| fun | fun | 1 | cut |
"strawberry" has 3 letter r's, but the model reads str | aw | berry; two pieces contain an r and none says how many. Tokenization is one reason letter counting is hard for models. It is not the only one.
- To do: Using the toy table (the=1 token, strawberry=3, tokenization=2, is=1, fun=1), how many tokens is "the strawberry tokenization is fun" (5 words)?
- To do: A budget of 50 tokens: how many complete copies of that sentence fit?
- To do: "strawberry" is seen as str | aw | berry. The word has 3 letter r's. How many of the 3 pieces contain at least one r?
Part 2. Do it in code
- token_count(words, table): table maps a word to its list of pieces. Sum the pieces for all words. A word missing from the table counts as one piece per character.
- copies_that_fit(words, table, budget): how many complete copies of the word list fit in the budget (0 if one copy does not fit; ValueError if the word list needs 0 tokens).
- prefix_within_budget(words, table, budget): the longest prefix of the word list whose token count is at most the budget.
- pieces_with_letter(word, table, letter): return (letters, pieces): how many times letter occurs in the word, and how many of the word's pieces contain it.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does a token budget guarantee a fixed number of visible words?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
In a toy merge, replacing m nonoverlapping adjacent pairs by one symbol reduces sequence length by m. A request budget must include formatted input plus requested output within the model's supported context; exact accounting is model-specific.
Worked example
Start with toy symbols ['a','b','a','b'] and a vocabulary containing a and b. The pair (a,b) occurs twice. A merge creates symbol ab and encodes the sequence as ['ab','ab'], reducing length from 4 to 2. This deliberately omits byte encoding and pre-tokenization so the merge operation is visible.
Reference code
The following example uses the Python standard library and does not call a remote service.
tokens = list('abab')
merged=[]
i=0
while i < len(tokens):
if tokens[i:i+2] == ['a','b']:
merged.append('ab'); i += 2
else:
merged.append(tokens[i]); i += 1
assert merged == ['ab','ab']
assert len(tokens)-len(merged) == 2
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A token is not necessarily a word. A vocabulary ID is a label, not a magnitude. A hand-built toy merge sequence is not a drop-in replacement for a production tokenizer.