LLMs

Trace a toy tokenizer and explain byte-pair merges, vocabulary IDs and token-budget limits.

1. L1.1Characters, bytes and tokens
2. L1.2Vocabulary IDs
3. L1.3Byte-pair merging
4. L1.4Encoding and decoding
5. L1.5Special tokens and templates
6. L1.6Token budgets and limitations

Perform an embedding lookup and explain similarity, positional information and rotary position encoding.

1. L2.1Embedding lookup
2. L2.2Similarity and context
3. L2.3Position information
4. L2.4Rotations in two dimensions
5. L2.5Rotary position encoding
6. L2.6Geometry and evaluation

Build a smoothed bigram model and connect next-token probabilities to log loss and perplexity.

1. L3.1Next-token targets
2. L3.2Counting bigrams
3. L3.3Smoothing
4. L3.4Log loss
5. L3.5Sampling
6. L3.6Beyond bigrams

Calculate query-key scores, scaled softmax weights and the weighted value output for one attention head.

1. L4.1Queries, keys and values
2. L4.2Dot-product scores
3. L4.3Scaling
4. L4.4Softmax weights
5. L4.5Weighted values
6. L4.6Masking and multiple heads

Trace a transformer block, distinguish normalization variants and count a simplified block's parameters.

1. L5.1Residual streams
2. L5.2LayerNorm and RMSNorm
3. L5.3The feed-forward sublayer
4. L5.4Pre-norm and post-norm
5. L5.5MQA and GQA
6. L5.6Parameter counting

Describe a pretraining loop and reason about data, batch size, precision and a simplified compute budget.

1. L6.1Training data
2. L6.2Targets and loss
3. L6.3Batches and accumulation
4. L6.4Mixed precision
5. L6.5Scaling and budgets
6. L6.6Parallel training

Compare supervised fine-tuning, preference optimization and parameter-efficient adaptation without conflating their objectives.

1. L7.1Supervised fine-tuning
2. L7.2Human preferences and rewards
3. L7.3RLHF and constraints
4. L7.4Direct preference optimization
5. L7.5LoRA and QLoRA
6. L7.6Distillation and reasoning training

Compute temperature and candidate filtering, and distinguish sampling from greedy and beam decoding.

1. L8.1Greedy selection
2. L8.2Temperature
3. L8.3Top-k
4. L8.4Top-p
5. L8.5Beam search
6. L8.6Constraints and repetition

Explain expert routing, long-context costs, state-space models and multimodal token streams under explicit assumptions.

1. L9.1Mixture of experts
2. L9.2Routing and load balance
3. L9.3Long context
4. L9.4State-space models
5. L9.5Multimodal representations
6. L9.6Comparing architecture claims

Assemble a tiny causal language-model experiment with a clear data split, baseline, training loop and evaluation.

1. L10.1Data and vocabulary
2. L10.2A baseline
3. L10.3A causal block
4. L10.4Optimization
5. L10.5Sampling and evaluation
6. L10.6An experiment report