Axis: attention spine (sibling of Modules 3, 4, 6) · Prereq: Module 3 (the ), Module 4 () · Next: Module 6 (frontier ) Hook: and MLA shrank the KV-cache but attention is still O(n²) in compute and its state still grows with context. Linear attention and state-space models change the math itself: O(n) sequence mixing with a fixed-size state — no per-token KV growth at all. Attacks cost #2 (compute) and cost #3 (KV memory) by deleting the quadratic — at a real cost to exact recall.
The 2-minute version (no math)
Softmax attention (Modules 1–4) re-reads the entire history to produce each new token, and stores a Key/Value for every past token — so both the work and the memory grow with the length of the conversation.
Linear attention and SSMs (state-space models, e.g. ) work more like an old-school RNN: they keep a small, fixed-size running summary ("state") and roll it forward one token at a time. The cost per token is constant — it doesn't matter whether you're 1,000 or 1,000,000 tokens in. That's a huge win for long context.
The catch: a fixed-size summary can only remember so much — it forgets, and can't reach back and attend to an arbitrary earlier token the way real attention can. So modern models don't go pure — they interleave a few full-attention layers among many linear/SSM layers, getting most of the speed while keeping enough exact recall to stay sharp.
Under the hood
- Softmax attention computes
softmax(QKᵀ)V: theQKᵀterm is O(n²) and forces a KV-cache that grows withn. - Linear attention replaces the softmax with a kernel feature map
φ, so the computation can be re-associated asφ(Q)·(φ(K)ᵀV). At decode time this accumulates into a state whose size is fixed w.r.t. context length (ad×d-scale matrix per layer) and is updated recurrently — O(n) time, and a state that doesn't grow with n (no growing KV-cache). Two honest caveats: "fixed" ≠ "free" — that matrix can be millions of floats per layer (≈d²), and at short context it may even exceed a small KV-cache; and this neat recurrence is the decode view — training uses a parallel/chunked form to stay tractable, which is not O(1) in memory. - SSM / Mamba is a selective state-space recurrence: a learned linear dynamical system with input-dependent that decides what to keep in its fixed hidden state. Also O(n), fixed state, computed with a parallel "selective scan."
- The trade is exact recall. A fixed state is a lossy summary; neither can perfectly retrieve an
arbitrary past token (the thing softmax attention is great at). So the frontier answer is a
hybrid stack: many cheap linear/SSM layers + a few full-attention layers. Those full-attention
layers act as periodic exact-recall checkpoints — able to attend over the entire history, they
re-inject what the fixed-state layers compressed away, so the stack keeps precise long-range lookup
while paying the O(n) cost most of the time. The
layer_types/ block partition in the config declares the pattern — the topology view we return to in Module 10.
Fingerprint evidence
(auto-populated — _evidence/module-05.evidence.md)
| Model | Layer layout | Also carries |
|---|---|---|
Qwen3.8-2.4T-A95B | 69× linear-attention / 23× full-attention (≈3:1) | , |
NVIDIA-Nemotron-3-Ultra-550B | 48× Mamba-SSM / 48× MoE / 12× attention (is_hybrid_ssm) | MoE, MTP |
Across the atlas: 69 models are SSM-hybrids (is_hybrid_ssm) and 27 carry the explicit
hybrid_linear_attention tag. Note nobody ships pure linear/SSM at frontier scale — the layouts
above always keep a slice of real attention. That ratio (linear-or-SSM : full-attention) is a key
knob — but not the whole story: where the attention layers sit (early, late, evenly interleaved)
matters as much as how many, and whether the bet actually pays off is a measured () recall
question, not something the ratio alone tells you.
The honest trade-off (the Verdict)
- Which cost: buys cost #2 (O(n) compute) and cost #3 (fixed state → no KV growth) — the only techniques in the guide that attack the quadratic itself rather than shrinking its constant.
- What it trades: exact recall. That's why every real deployment is a hybrid; a pure linear/SSM stack degrades on tasks needing precise long-range lookup. The layer ratio is the lab's bet on how much recall it can afford to approximate.
- / "active" get fuzzy here (guardrail): in an SSM layer there's no discrete "which fired" set, so parameter-usage ratios don't map cleanly — don't force an SR reading onto a hybrid.
- Tier discipline: the layer layout and tag are Tier-2 (config). Whether a given hybrid holds up on long-context recall is Tier-1 (measured) — the config tells you the bet, not the outcome.
Glossary delta
linear attention · SSM (state-space model) / Mamba · recurrent / fixed state · sequence mixing · selective scan · hybrid layer ratio · kernel feature map
Prev: Module 4 — Latent attention (MLA) · Next: Module 6 — Frontier attention sparsity () Evidence: atlas snapshot 2026-08-13. Exemplars: Qwen3.8, Nemotron-3-Ultra.