Axis: attention spine (the frontier rung; sibling of Modules 3–5) · Prereq: Module 3 (full attention), Module 5 (why fixed-state models lose recall) · Next: Module 7 () Hook: Linear/ (Module 5) killed the quadratic but gave up exact recall. Frontier — (DSA) and kin — keeps softmax attention's exact recall while making it sub-quadratic, by attending to only a selected subset of past tokens per query. Attacks cost #2 (compute) with far less quality loss than a fixed state.
The 2-minute version (no math)
Full attention re-reads every past token for every new one — precise, but expensive (Module 3). Module 5's linear/SSM trick made it cheap but forgetful.
DSA takes a third road: keep real attention, but for each new token learn which few past tokens actually matter and attend to just those — like skimming a long document straight to the relevant lines instead of re-reading all of it. You keep the precision of true attention and drop most of the cost. The price is a small learned "selector" that decides what to look at, plus fiddly GPU kernels to make the skipping fast.
This is attention sparsity — skipping token-pairs. Don't confuse it with the parameter sparsity of Mixture-of-Experts (Module 7), which skips weights. Different axis, different word.
Under the hood
- DSA (DeepSeek Sparse Attention): a lightweight indexer scores past tokens for each query and selects the most relevant; attention runs only over that subset — able to reach an arbitrary far-back token (unlike 's fixed local window, and unlike a lossy fixed state). The selection is content-based rather than positional — but honestly, the indexer is a cheap surrogate for relevance (often a low-rank projection), not full attention's judgment; it can miss a token that matters only in combination with another.
- "Sub-quadratic" has a condition worth stating: it holds only if the selector itself is approximate/indexed (block-level, hashed, or clustered scoring). A brute-force selector that scored every past token for every query would still be O(n²) — the win comes from not doing that. So DSA is cheap conditionally, on a good index, not unconditionally.
- Attention sinks — a few tokens (often the first) that every query always attends to — stabilize the distribution so the sparse selection doesn't collapse.
- vs SWA (Module 3): SWA always keeps a fixed local window; DSA chooses which tokens, and can choose distant ones. vs linear/SSM (Module 5): those approximate all history in a fixed state; DSA runs true softmax attention over the chosen subset — each kept pair's score is exact, though the softmax normalizes over the subset, not all tokens, so it isn't identical to full attention (hence "exact-ish").
- vs MoE (Module 7) — the terminology guard: MoE is parameter sparsity (most weights idle per token). DSA is attention sparsity (most token-pairs skipped). We always qualify which.
Fingerprint evidence
(auto-populated — _evidence/module-06.evidence.md)
| Model | Frontier-attention technique | Note |
|---|---|---|
DeepSeek-V4-Pro-0813 | DSA (+ + ) | the origin line |
GLM-5.2-FP8 | DSA (+ MLA) | DSA propagated to an adopter |
MiniMax-M3 | sparse_attention | an independent frontier-attention design (not DSA) |
DSA appears on 24 current models — concentrated in the DeepSeek lineage and its adopters
(GLM), not spread evenly. That concentration is itself intelligence: like MLA (Module 4), DSA is a
DeepSeek-origin technique whose propagation we can track — provenance, not just capability.
MiniMax-M3's independent sparse_attention shows the idea (attention sparsity) is being reinvented
separately, which is a different signal from adopting DeepSeek's specific mechanism.
The honest trade-off (the Verdict)
- Which cost: buys cost #2 (compute → sub-quadratic) while keeping exact-ish recall — arguably the "best of both" against Module 5's fixed-state approximation. Cost #3 (KV memory) still needs a companion (MLA/); DSA is about what you compute over, not what you cache.
- What it trades: a learned selector (extra complexity, extra kernels) and the risk that the selection mis-fires on an adversarial layout — harder to get right than a fixed window.
- Provenance over merit (guardrail): the 24-model concentration is an adoption signal (DeepSeek-origin, propagating), not a measured claim that DSA is best — that would need ablations. Whether DSA-at-length beats a hybrid-SSM is and not read from the config.
Glossary delta
attention sparsity (vs ) · DSA (DeepSeek Sparse Attention) · token selector / indexer · top-k token selection · attention sinks · content-based vs positional sparsity
Prev: Module 5 — Linear attention & SSM · Next: Module 7 — Parameter sparsity: Mixture-of-Experts Evidence: atlas snapshot 2026-08-13. Exemplars: DeepSeek-V4-Pro, GLM-5.2, MiniMax-M3.