L9.6 · LESSON 6 OF 6

Comparing architecture claims

Explain expert routing, long-context costs, state-space models and multimodal token streams under explicit assumptions.

THE IDEA

Reading architecture claims

Claims such as "faster" or "bigger" hide the details that decide what is true for you. Record, at least: total parameters, active parameters per token, sequence length, the quality target and how it is measured, the hardware, and the workload. Memory follows the total size; compute per token follows the active size.

model	total parameters	active per token	memory (2 bytes each)	~FLOPs per token (2 × active)
A: dense	7,000,000,000	7,000,000,000	14 GB	14,000,000,000
B: mixture of experts	10,000,000,000	4,000,000,000	20 GB	8,000,000,000

B is bigger to store (20 GB vs 14 GB) but cheaper to run per token (8e9 vs 14e9 FLOPs). Neither number alone says which is faster in your deployment: speed also depends on hardware, batch size, expert routing and context length.

When you read an architecture claim, look for: total size, active size, sequence length, the quality target and how it was measured, the hardware, and the workload. A single speed number cannot summarise every setting.

A dense model and a mixture-of-experts model compared on two axes.

Predict first. Can a larger total model have fewer active expert weights per token?

Chapter context

New architectures alter where computation and memory are spent. Compare them by tracing one request through their actual operations rather than assuming every improvement reduces every cost.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out memory and compute for both models by hand, then write a claim checker that flags missing information.

model	total parameters	active per token	memory (2 bytes each)	~FLOPs per token (2 × active)
A: dense	7,000,000,000	7,000,000,000	14 GB	14,000,000,000
B: mixture of experts	10,000,000,000	4,000,000,000	20 GB	8,000,000,000

GB to store model B billions of FLOPs per token for B fewer parameters per token (A or B)

To do: Model B (the mixture-of-experts model: 10 billion total, 4 billion active) is stored with 2 bytes per parameter. How many GB (10⁹ bytes) must be stored?
To do: A common rough estimate is 2 FLOPs per ACTIVE parameter per generated token. For model B (4 billion active), FLOPs per token in billions?
To do: Model A is dense with 7 billion parameters. Which model computes with FEWER parameters per token, A or B? Type the letter

Part 2. Do it in code

summarize(model): model is a dict with total_params, active_params and bytes_per_param. Return a dict with total_gb (total·bytes/1e9), flops_per_token (2·active) and active_fraction (active/total).
compare(a, b): both are summarize results. Return a dict with memory_ratio (a total_gb / b total_gb) and compute_ratio (a flops / b flops).
missing_fields(claim): a claim is a dict. The required fields are total_params, active_params, seq_len, hardware, workload and quality_metric. Return the sorted list of required fields that are missing or None.
is_comparable(claim_a, claim_b): True only if neither claim has missing fields and both use the same hardware and workload.

def summarize(model):
    pass

def compare(a, b):
    pass

def missing_fields(claim):
    pass

def is_comparable(claim_a, claim_b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a larger total model have fewer active expert weights per token?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a toy MoE with shared weights S, E equal experts of P weights and k experts active per token:

N_{total}=S+EP,\quad N_{active}\approx S+kP.

This is a parameter-accounting model, not a full FLOP or communication model. Naively materialized attention scores use $T^2$ entries per head.

Worked example

With 2 billion shared parameters and eight experts of 1 billion each, total parameters are 10 billion. If two experts are selected per token, approximately 4 billion parameters participate along that token's path. The other experts still require storage or loading. A 4096×4096 naive score matrix contains 16,777,216 entries per head.

Reference code

The following example uses the Python standard library and does not call a remote service.

shared, experts, per_expert, active=2,8,1,2
assert shared+experts*per_expert == 10
assert shared+active*per_expert == 4
assert 4096**2 == 16777216

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Active parameters and resident parameters differ. Linear sequence complexity does not make every model property linear. Context-window capacity and demonstrated context use are separate claims.