Encoding and decoding
Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.
THE IDEA
Encoding replays the merges; decoding is just joining
To encode new text with a trained tokenizer, split it into the starting symbols and apply the learned merges in the order they were learned. Decoding concatenates the pieces back. Two things follow: the order of merges matters, and the IDs only mean something together with the exact vocabulary that produced them.
Encoding replays the learned merges in their learned order. Decoding just joins the pieces. The same ID list means different text under a different vocabulary:
Predict first. Should tokens from one tokenizer be decoded with an unrelated vocabulary?
Chapter context
A language model receives token IDs, not visible words. The tokenizer chooses how text becomes those IDs. Different tokenizers can represent the same string with different lengths, so character count is not a universal token count.
TRY IT YOURSELF
Part 1. Do it hands-on
Encode a few words by hand, spot the vocabulary mix-up, then write the encoder and decoder.
Encoding replays the learned merges in their learned order. Decoding just joins the pieces. The same ID list means different text under a different vocabulary:
- To do: Merges, in order: (e,s) then (es,t). Encode "newest". How many tokens?
- To do: How many tokens does "west" become?
- To do: How many tokens does "set" become? (the letters are s, e, t)
- To do: Token IDs [0, 1] were produced with vocabulary A (0 = a, 1 = b) but decoded with vocabulary B (0 = b, 1 = a). Type the text that comes out
Part 2. Do it in code
- apply_merges(word, merges): start from the list of letters and apply each merge pair in order (each pair replaces every non-overlapping adjacent occurrence, left to right). Return the list of tokens.
- encode_text(text, merges): split text on spaces, apply_merges to each word, and return one flat list of tokens, putting the string "_" after each word except the last as a separator token.
- decode_tokens(tokens): join the tokens and turn each "_" back into a space.
- decode_ids(ids, id_to_token): map ids to tokens with the table and join them. Raise ValueError for an unknown id.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Should tokens from one tokenizer be decoded with an unrelated vocabulary?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
In a toy merge, replacing m nonoverlapping adjacent pairs by one symbol reduces sequence length by m. A request budget must include formatted input plus requested output within the model's supported context; exact accounting is model-specific.
Worked example
Start with toy symbols ['a','b','a','b'] and a vocabulary containing a and b. The pair (a,b) occurs twice. A merge creates symbol ab and encodes the sequence as ['ab','ab'], reducing length from 4 to 2. This deliberately omits byte encoding and pre-tokenization so the merge operation is visible.
Reference code
The following example uses the Python standard library and does not call a remote service.
tokens = list('abab')
merged=[]
i=0
while i < len(tokens):
if tokens[i:i+2] == ['a','b']:
merged.append('ab'); i += 2
else:
merged.append(tokens[i]); i += 1
assert merged == ['ab','ab']
assert len(tokens)-len(merged) == 2
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A token is not necessarily a word. A vocabulary ID is a label, not a magnitude. A hand-built toy merge sequence is not a drop-in replacement for a production tokenizer.