L1.1 · LESSON 1 OF 6

Characters, bytes and tokens

Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.

THE IDEA

Three different things people call "a letter"

A language model cannot read text directly; it reads numbers. The first question is: numbers for what? There are three different units, and mixing them up is the source of many surprises.

A character (more precisely a Unicode code point) is a symbol such as A, é or 😀.
A byte is a number from 0 to 255. Text is stored as bytes using an encoding; UTF-8 uses 1 byte for ASCII letters and up to 4 for other symbols.
A token is whatever piece the model's tokenizer decides to treat as one unit: a byte, a word piece, or a whole word.

type any text

character	code point	UTF-8 bytes	count
h	U+0068	68	1
é	U+00E9	C3 A9	2
l	U+006C	6C	1
l	U+006C	6C	1
o	U+006F	6F	1
😀	U+1F600	F0 9F 98 80	4

6 characters (code points) → 10 bytes A byte-level tokenizer starts from these 10 numbers (each 0–255). Token count depends on the tokenizer on top.

Type text to see its characters and bytes. Try an accented letter, an emoji and a family emoji.

Predict first. Must an emoji occupy exactly one byte and one token?

Python

"héllo😀".encode("utf-8")   # b'h\xc3\xa9llo\xf0\x9f\x98\x80' -> 10 bytes for 6 characters

Chapter context

A language model receives token IDs, not visible words. The tokenizer chooses how text becomes those IDs. Different tokenizers can represent the same string with different lengths, so character count is not a universal token count.

TRY IT YOURSELF

Part 1. Do it hands-on

Predict byte counts using the table, then write functions that convert text to bytes and back.

type any text

character	code point	UTF-8 bytes	count
h	U+0068	68	1
é	U+00E9	C3 A9	2
l	U+006C	6C	1
l	U+006C	6C	1
o	U+006F	6F	1
😀	U+1F600	F0 9F 98 80	4

6 characters (code points) → 10 bytes A byte-level tokenizer starts from these 10 numbers (each 0–255). Token count depends on the tokenizer on top.

bytes for A, é, €, 😀:

"héllo😀": charactersbytes

family emoji: code pointsbytes

To do: How many UTF-8 bytes does each character need: A, é, €, 😀? Type four numbers
To do: The text "héllo😀": how many characters, and how many UTF-8 bytes?
To do: The family emoji 👨‍👩‍👧 is one symbol on screen. It is built from three people joined by two invisible joiners. How many code points, and how many UTF-8 bytes?

Part 2. Do it in code

utf8_bytes(text): the UTF-8 bytes of the text as a list of integers.
char_count(text): the number of Unicode code points (len of the string).
byte_tokens(text): the byte-level tokens: the same list as utf8_bytes (every byte is a token in the range 0-255).
from_byte_tokens(tokens): turn byte tokens back into text with UTF-8 decoding. Raise ValueError if a token is not in 0-255 or the bytes are not valid UTF-8.
bytes_per_char(text): the average number of bytes per character (a float). Raise ValueError for an empty text.

def utf8_bytes(text):
    pass

def char_count(text):
    pass

def byte_tokens(text):
    pass

def from_byte_tokens(tokens):
    pass

def bytes_per_char(text):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must an emoji occupy exactly one byte and one token?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

In a toy merge, replacing m nonoverlapping adjacent pairs by one symbol reduces sequence length by m. A request budget must include formatted input plus requested output within the model's supported context; exact accounting is model-specific.

Worked example

Start with toy symbols ['a','b','a','b'] and a vocabulary containing a and b. The pair (a,b) occurs twice. A merge creates symbol ab and encodes the sequence as ['ab','ab'], reducing length from 4 to 2. This deliberately omits byte encoding and pre-tokenization so the merge operation is visible.

Reference code

The following example uses the Python standard library and does not call a remote service.

tokens = list('abab')
merged=[]
i=0
while i < len(tokens):
    if tokens[i:i+2] == ['a','b']:
        merged.append('ab'); i += 2
    else:
        merged.append(tokens[i]); i += 1
assert merged == ['ab','ab']
assert len(tokens)-len(merged) == 2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A token is not necessarily a word. A vocabulary ID is a label, not a magnitude. A hand-built toy merge sequence is not a drop-in replacement for a production tokenizer.