Axis: — (foundation) · Prereq: none · Next: Module 1 (the dense baseline) Hook: Before you can understand a single efficiency technique, you need the ruler that measures them. "Efficiency" is not one thing — it's five. This module installs that ruler; every later module names which of the five it buys and which it trades.
The 2-minute version (no math)
A language model is a next-token predictor: feed it text, it predicts the next word-piece, append it, repeat. Under the hood it's one small machine — a block — stacked many times (tens of them), plus a big lookup table (the vocabulary embedding) at each end.
When people say a model is "efficient," they can mean five different things, and mixing them up is the single most common way to misread a model:
- How big is it on disk? (parameter count)
- How much math per word? (compute / FLOPs)
- How much memory does a long conversation eat? ()
- How fast does it answer? (latency)
- How much does it cost to run? (a blend of the above)
A trick that makes one better can make another worse. A "" model, for instance, does far less math per word — but it's bigger on disk and needs more fast GPU memory, not less. So there's never a single "is it efficient?" answer. There's "efficient at what?" Keep the five in mind and every model in our atlas becomes readable.
Under the hood
What "a model" actually is, at the file level. Two things: a config.json (the architecture —
how many layers, how wide, which attention type, how many …) and a set of weight tensors
(the learned numbers). ModelDNA's config-only fingerprint reads the first; weight-level analysis
reads the second. Everything this guide teaches is a choice recorded in that config and realized in
those weights.
First, one term used constantly: a block (or layer) is one instance of the repeated transformer unit — an attention sub-layer plus an sub-layer, each with its norm and . "32 layers" means that unit stacked 32 times.
The one distinction the whole guide turns on: total vs active parameters.
- In a dense model, every parameter is used for every token.
active = total. - In a sparse (MoE) model, a sends each token through only a few of many "expert"
sub-networks, so only a fraction of the weights do work on any given token.
active ≪ total. Strictly, = the always-on core (embeddings, attention, norms, any ) + thekrouted experts chosen for this token + the router itself. It is a per-token quantity.
The ratio = total ÷ active is the headline number for how sparse a model is — but (guardrail, see Verdict) it is a parameter-usage ratio, a headline, not a verdict on speed or cost.
The ruler: four technical costs, and the bill they compose into. Each technique in this guide attacks one or two of the four independent costs. The fifth line — serving cost — is not a peer axis; it's the composite the other four roll up into (flagged as such so we don't pretend it's a separate lever).
| # | Cost | Precisely | Who pays it | Attacked by |
|---|---|---|---|---|
| 1 | Parameter count | total weights stored | disk, training compute, VRAM to hold the model | (MoE raises this) |
| 2 | Compute / | arithmetic per generated token | throughput, $/token, energy | MoE (active↓), linear attn, |
| 3 | KV-cache memory | attention state kept per sequence, grows with context length | long-context VRAM — often the real wall | , , , |
| 4 | Decode latency | wall-clock per token | interactivity, tokens/sec | / speculative decoding |
| — | Serving cost (composite) | $/token under a fixed utilization model — derived from 1–4, not independent | your GPU bill | nothing directly; it's the sum of the trades above |
Note the difference between #1 and #3: parameters are static (fixed the moment you load the model); KV-cache is dynamic (zero at the first token, growing with every token of context). They are different memories with different growth laws — conflating them is a classic error.
The key insight that makes the ruler necessary: the four costs trade against each other. MoE cuts cost #2 (FLOPs-per-token) dramatically, but raises cost #1 (parameter count) and pressures VRAM — because every expert's weights must sit in fast GPU memory ready to be fetched, even the ones idle on this token. So "MoE is efficient" is only true for cost #2. State which cost, or you've said nothing.
This guide is a taxonomy, not a ranking — no upgrade path is implied. The modules that follow (the attention spine — GQA, MLA, linear/SSM, DSA — plus the orthogonal axes of MoE, MTP, and precision) are different trades against the five costs above, not a ladder from worse to better. "MLA beats GQA on KV-cache" does not mean MLA is a better choice everywhere — it's a different point on the cost surface, with its own price (see each module's Verdict). Phrases like "the next rung" order the reading, not the techniques. A frontier model mixes these axes; it does not "graduate" up a single line. All of this is structure (what a model is), never a measured claim that one technique is superior — that would need ablations we don't have.
Fingerprint evidence — same word, opposite cost profiles
Two atlas models, both correctly called "a model," with radically different rulers:
| Model | Total params | Active params | SR | Reading (a parameter-usage ratio only) |
|---|---|---|---|---|
Llama-3.1-8B (dense) | 8.0 B | 8.0 B | 1× | active = total, so SR = 1 by arithmetic (not "convention") |
Ling-3.0-flash (MoE) | 129 B | 2.8 B | 45× | only ~1/45 of the parameters are used per token — SR is about which weights work, not a 45× speed/cost claim (that depends on expert width + routing; Module 7) |
Ling-3.0-flash needs the VRAM of a 129 B model to hold (cost #1, big) while using far fewer
parameters per token (a low active count, which tends to lower cost #2 — by how much depends on
expert width and routing overhead, not on SR alone). Cost #1 and cost #2 point in opposite
directions in the same model. That is the entire reason the ruler exists — and it's why the rest of
this guide is organized by which cost each technique attacks, not by a single "efficiency" score.
The honest trade-off (the Verdict)
- There is no scalar "efficiency." Any single number ("SR = 45×", "8 B params") describes one of five costs. We always say which.
- SR is a headline, not a verdict (pre-committed guardrail). It tells you the sparsity of the parameter usage; it does not tell you speed or serving cost, because it ignores the memory- bandwidth reality of cost #5. A 45× SR model is not 45× cheaper to serve. Where we have a measured latency or tokens/sec, we pair it with SR; where we don't, we say so.
- "Tested" means two things (guardrail): breadth (~2,900 models) is config-fingerprinted (Tier-2 structure); a subset also has measured behavior (Tier-1: ARS, worldview, claims — dated). This module is Tier-2 throughout.
Glossary delta
token · block / layer · parameter (total vs active) · Sparsity Ratio (SR) · config.json / weights · KV-cache · FLOPs-per-token · the five efficiencies
Prev: — · Next: Module 1 — The dense baseline (and its atoms) Evidence: atlas snapshot 2026-08-13. Exemplars: Llama-3.1-8B, Ling-3.0-flash.