L9.5 · LESSON 5 OF 6

Multimodal representations

Explain expert routing, long-context costs, state-space models and multimodal token streams under explicit assumptions.

THE IDEA

Images and audio as token streams

To feed images or sound to a transformer, a modality-specific encoder turns them into a sequence of vectors that can sit beside text tokens. A common image recipe cuts the picture into square patches; a 224×224 image in 16×16 patches gives 14 × 14 = 196 vectors. Audio is cut into overlapping frames. How many tokens result depends on the architecture.

image side (pixels)224patch size16images3text tokens40

patches per image = (224/16)² = 196 sequence length = 3 × 196 + 40 = 628 tokens audio: 1 s at 16 kHz, window 400, hop 160 → 98 frames

An image is not one token: how many it becomes depends on the encoder (patch size, any pooling or resizing). The same holds for audio frame rates.

Change the patch size and image count.

Predict first. Is one image always one model token?

Chapter context

New architectures alter where computation and memory are spent. Compare them by tracing one request through their actual operations rather than assuming every improvement reduces every cost.

TRY IT YOURSELF

Part 1. Do it hands-on

Count patches, frames and the combined sequence length, then write a patchify function.

image side (pixels)224patch size16images3text tokens40

patches per image = (224/16)² = 196 sequence length = 3 × 196 + 40 = 628 tokens audio: 1 s at 16 kHz, window 400, hop 160 → 98 frames

An image is not one token: how many it becomes depends on the encoder (patch size, any pooling or resizing). The same holds for audio frame rates.

patches in 224×224 at 16×16 3 images + 40 text tokens audio frames in one second

To do: An image encoder cuts a 224×224 image into 16×16 patches. How many patches?
To do: A prompt has 3 such images (one token per patch) and 40 text tokens. Total sequence length?
To do: One second of 16,000-sample audio, analysed in windows of 400 samples every 160 samples: frames = floor((16000 − 400) / 160) + 1. Type it

Part 2. Do it in code

patch_count(h, w, p): number of non-overlapping p×p patches (ignore leftovers): (h // p) · (w // p).
patchify(img, p): img has shape (H, W, C). Return an array (n_patches, p·p·C), patches in reading order (row by row), each flattened. H and W must be multiples of p (ValueError).
audio_frames(samples, win, hop): floor((samples − win) / hop) + 1, or 0 if samples < win.
sequence_length(n_images, tokens_per_image, text_tokens): total length of the combined token stream.

import numpy as np

def patch_count(h, w, p):
    pass

def patchify(img, p):
    pass

def audio_frames(samples, win, hop):
    pass

def sequence_length(n_images, tokens_per_image, text_tokens):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is one image always one model token?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a toy MoE with shared weights S, E equal experts of P weights and k experts active per token:

N_{total}=S+EP,\quad N_{active}\approx S+kP.

This is a parameter-accounting model, not a full FLOP or communication model. Naively materialized attention scores use $T^2$ entries per head.

Worked example

With 2 billion shared parameters and eight experts of 1 billion each, total parameters are 10 billion. If two experts are selected per token, approximately 4 billion parameters participate along that token's path. The other experts still require storage or loading. A 4096×4096 naive score matrix contains 16,777,216 entries per head.

Reference code

The following example uses the Python standard library and does not call a remote service.

shared, experts, per_expert, active=2,8,1,2
assert shared+experts*per_expert == 10
assert shared+active*per_expert == 4
assert 4096**2 == 16777216

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Active parameters and resident parameters differ. Linear sequence complexity does not make every model property linear. Context-window capacity and demonstrated context use are separate claims.