L10.1 · LESSON 1 OF 6

Data and vocabulary

Assemble a tiny causal language-model experiment with a clear data split, baseline, training loop and evaluation.

THE IDEA

Capstone: one small experiment, end to end

This chapter joins the earlier pieces into a tiny, inspectable character-level language model. The aim is to understand the model and the evidence behind it, not to compete with production assistants. Each lesson is one stage of the experiment and each exercise builds that stage in Python.

Stage one is the data. Use text you have permission to use, turn characters into ids with a vocabulary, and split before you window: validation text must never share tokens with training windows.

training text

training share0.8window length T8

to be or not to be that is the question

39 characters → 31 training (blue) + 8 validation (orange), split BEFORE windowing windows: training 23, validation 0 (a part of n tokens gives n − T) if windowed first: 31 windows, 8 of which straddle the cut vocabulary (13): ␣ a b e h i n o q r s t u

Save the vocabulary (the exact id ↔ character table) next to the model: the learned embedding rows belong to those ids, and a different mapping would silently scramble them.

Type your own text. Notice how many windows each part yields, and how many would straddle the cut.

Predict first. Why must the vocabulary mapping survive model reload?

Chapter context

The capstone joins the earlier pieces into one inspectable experiment. The goal is to understand the model and its evidence, not to claim that a toy trained on a few paragraphs competes with a production assistant.

TRY IT YOURSELF

Part 1. Do it hands-on

Count windows for a clean split, then write the vocabulary, the split, the windowing and the save/load round trip.

training text

training share0.8window length T8

to be or not to be that is the question

Save the vocabulary (the exact id ↔ character table) next to the model: the learned embedding rows belong to those ids, and a different mapping would silently scramble them.

training windows (320 tokens, T = 8) validation windows (80) windows if built from all 400 first vocabulary of "hello"

To do: 400 tokens are split into 320 training and 80 validation tokens BEFORE windowing. With T = 8 each example is T inputs plus the shifted targets, so a part of n tokens gives n − T windows. Training windows?
To do: Validation windows?
To do: If you built windows from all 400 tokens first and split the windows afterwards, there are 400 − 8 windows. Type that number. Windows near the boundary mix training and validation tokens
To do: A character-level vocabulary for the text "hello" has how many entries?

Part 2. Do it in code

build_vocab(text): return (stoi, itos): the sorted unique characters, itos a list, stoi the dict char → index.
encode(text, stoi) → list of ids; decode(ids, itos) → string. Unknown characters raise ValueError.
split_tokens(tokens, train_frac): train = first int(len·train_frac) tokens, validation = the rest. Split BEFORE windowing. Raise ValueError if either part would have 2 tokens or fewer.
window_xy(tokens, T): arrays X, Y of shape (n − T, T): X[i] = tokens[i:i+T], Y[i] = tokens[i+1:i+T+1]. ValueError if there are not more than T tokens.
save_vocab(itos) → JSON string, load_vocab(s) → (stoi, itos): the vocabulary must survive a reload exactly.

import json
import numpy as np

def build_vocab(text):
    pass

def encode(text, stoi):
    pass

def decode(ids, itos):
    pass

def split_tokens(tokens, train_frac):
    pass

def window_xy(tokens, T):
    pass

def save_vocab(itos):
    pass

def load_vocab(s):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why must the vocabulary mapping survive model reload?

AReloading regenerates the same IDs automatically.BIt makes training run faster.CBecause learned embedding rows belong to those exact token IDs, and a different mapping would attach every row to the wrong character.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a token sequence of length n and context length T, stride-one windows containing T inputs and their shifted targets require T+1 original tokens. The number of complete windows is max(0,n-T). Keep this counting convention distinct from nonoverlapping batching.

Worked example

A 400-token text is split into 320 training tokens and 80 validation tokens before windowing. With T=8, the training part contains 320−8=312 complete shifted windows, while validation contains 72. Constructing windows before the split would risk overlap across the boundary.

Reference code

The following example uses the Python standard library and does not call a remote service.

text='abcd'*100
train,validation=text[:320],text[320:]
context=8
windows=[train[i:i+context+1] for i in range(len(train)-context)]
assert len(text) == 400
assert len(windows) == 312
assert all(len(w)==9 for w in windows)
assert len(validation)-context == 72

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Toy output quality is not a benchmark claim. Splitting overlapping windows after construction leaks shared text. Save the tokenizer and architecture configuration, not only raw weights.