Vocabulary IDs
Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.
THE IDEA
A vocabulary is a lookup table
Once text is cut into pieces, each distinct piece gets an integer ID, and the table of all pieces is the vocabulary. A few IDs are reserved for special tokens such as padding and "unknown". Encoding looks pieces up; decoding looks IDs up in the reverse direction.
The tokenizer's vocabulary, built from "the cat sat the mat":
The IDs are just positions in the table. sat = 4 is not "bigger" than cat = 2 in any meaning; a later embedding table gives each ID its own learned vector. An unknown word falls back to ID 1 and its identity is lost.
Predict first. Does token ID 101 mean twice the concept of token ID 50?
Chapter context
A language model receives token IDs, not visible words. The tokenizer chooses how text becomes those IDs. Different tokenizers can represent the same string with different lengths, so character count is not a universal token count.
TRY IT YOURSELF
Part 1. Do it hands-on
Work out the vocabulary by hand, then build a vocabulary, an encoder and a decoder in Python.
The tokenizer's vocabulary, built from "the cat sat the mat":
The IDs are just positions in the table. sat = 4 is not "bigger" than cat = 2 in any meaning; a later embedding table gives each ID its own learned vector. An unknown word falls back to ID 1 and its identity is lost.
- To do: Vocabulary rule: <pad> = 0, <unk> = 1, then the unique words of "the cat sat the mat" in alphabetical order from 2. How many entries?
- To do: What is the ID of "sat"?
- To do: Encode "the cat ran". Type the three IDs separated by spaces
Part 2. Do it in code
- build_vocab(tokens, specials=("<pad>", "<unk>")): a dict token -> id. The specials get ids 0, 1, ...; then the unique tokens, sorted alphabetically, get the next ids. A token that is also a special is not added twice.
- encode(tokens, vocab): the list of ids. A token missing from the vocabulary becomes the id of "<unk>". Raise KeyError-like error (ValueError is fine) if the vocabulary has no "<unk>".
- decode(ids, vocab): the list of tokens. Raise ValueError for an id that is not in the vocabulary.
- unk_rate(tokens, vocab): the fraction of tokens that are unknown (0 for an empty list).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does token ID 101 mean twice the concept of token ID 50?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
In a toy merge, replacing m nonoverlapping adjacent pairs by one symbol reduces sequence length by m. A request budget must include formatted input plus requested output within the model's supported context; exact accounting is model-specific.
Worked example
Start with toy symbols ['a','b','a','b'] and a vocabulary containing a and b. The pair (a,b) occurs twice. A merge creates symbol ab and encodes the sequence as ['ab','ab'], reducing length from 4 to 2. This deliberately omits byte encoding and pre-tokenization so the merge operation is visible.
Reference code
The following example uses the Python standard library and does not call a remote service.
tokens = list('abab')
merged=[]
i=0
while i < len(tokens):
if tokens[i:i+2] == ['a','b']:
merged.append('ab'); i += 2
else:
merged.append(tokens[i]); i += 1
assert merged == ['ab','ab']
assert len(tokens)-len(merged) == 2
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A token is not necessarily a word. A vocabulary ID is a label, not a magnitude. A hand-built toy merge sequence is not a drop-in replacement for a production tokenizer.