Axis: attention spine (first rung; siblings are Modules 4–6) · Prereq: Module 1 (the ), Module 2 (context length) · Next: Module 4 () Hook: Two cheap, near-universal tricks aimed at the long-context wall from Module 1. shrinks the KV-cache directly (fewer KV heads → cost #3). caps the per-token attention work (a shorter window → cost #2) — and shrinks the cache too, but only if the serving stack evicts old tokens. Keeping those two effects straight is the whole subtlety of this module.
The 2-minute version (no math)
Module 1 left us with a problem: to keep generating, a model caches the Keys and Values of every past token, and that cache grows until it fills the GPU. Attention is where the cache lives, so attention is where you attack it. Two cheap wins, both nearly free in quality:
- Share keys and values across heads (GQA). In the original design every attention "head" keeps its own keys and values. But you don't need that many copies — let a group of query heads share one set of K/V. Share 4-to-1 and the cache shrinks 4×. This is so cheap that essentially every modern model does it.
- Only look back so far (SWA — sliding-window attention). Most tokens only need nearby context, not the whole document. So let most layers attend to just a window of recent tokens, and keep a few "global" layers that can see everything. The cache for the local layers stops growing once the window is full.
Both trade a sliver of modeling power for a big cut in memory — which is why they're the first thing every architecture reaches for.
Under the hood
The head-sharing spectrum: → GQA → . Recall from Module 1 that the KV-cache size is proportional to the number of KV heads (not query heads). So:
- MHA — every query head has its own K/V head (
kv_heads = heads). Maximum quality, maximum cache. - GQA (grouped-query attention) —
Gquery heads share each KV head, sokv_heads = heads / G. The cache shrinks by the factorG. This is the modern default. - MQA (multi-query attention) — the extreme:
kv_heads = 1. Smallest cache; less common in current fingerprints than GQA (whether its quality cost bites is model-specific and measured, not something read off the config).
Worked, tying back to Module 1 (Llama-3.1-8B, 32 query heads, head_dim 128, 32 layers, bf16):
Note it's the KV-cache (Keys and Values) that shrinks — queries are computed fresh each step and never cached, so GQA's win is purely on K/V. Per token, per layer (× the layer count for the model total):
| Scheme | KV heads | KV-cache / token / layer | × 32 layers = /token |
|---|---|---|---|
| MHA (hypothetical) | 32 | 2×32×128×2 B = 16,384 B | 512 KB |
| GQA 4:1 (actual) | 8 | 2×8×128×2 B = 4,096 B | 128 KB |
| MQA (hypothetical) | 1 | 2×1×128×2 B = 512 B | 16 KB |
GQA 4:1 is exactly why Llama-3.1-8B's KV-cache is 128 KB/token and not 512 KB — a 4× cut on the KV-cache for almost no quality loss. (This is the 128 KB/token number from Module 1, now explained.) And it scales with context: at a 128K-token window that 128 KB/token becomes ~16 GB — which is the real wall, and why the whole attention spine (this module + Module 4) exists.
Sliding-window attention (SWA). Independently of head-sharing, you can cap how far back attention
looks. A layer with window W attends only to the last W tokens. The clean, guaranteed win is on
compute: attention cost drops to O(W) per token instead of O(context). The memory win is
conditional — the KV-cache only falls to O(W) if the serving stack evicts tokens outside the
window with a rolling buffer (e.g. vLLM's sliding-window block manager, or a rotating-cache
kernel); many other implementations (Mistral-, Gemma-style) allocate the full-context cache and simply
mask attention, so storage stays O(context) even though compute is O(W). So SWA reliably buys cost
#2, and cost #3 only when the kernel actually evicts — don't assume the memory win is automatic. Pure local attention would lose
long-range reasoning, so models interleave: many cheap windowed layers plus a few global layers
(full, un-windowed attention over the whole context) that carry long-range information. That interleave
is what our fingerprint tags gqa_swa_hybrid.
fused_qkv is a different kind of efficiency worth not confusing with the above: it packs the
Q, K, V projections into one matrix multiply. That's a kernel/compute nicety (fewer, bigger matmuls) —
it does not change the KV-cache. Included because it shows up in the same fingerprints (e.g.
Qwen3.6-27B).
Fingerprint evidence
(auto-populated — _evidence/module-03.evidence.md)
A gradient of GQA aggressiveness, plus the SWA exemplar:
| Model | Query heads | KV heads | Group ratio | Note |
|---|---|---|---|---|
Llama-3.1-8B | 32 | 8 | 4:1 | the mainstream GQA setting |
Qwen3.6-27B | 24 | 4 | 6:1 | more aggressive; also fused_qkv |
gpt-oss-120b | 64 | 8 | 8:1 | aggressive KV reduction |
gemma-4-31B-it | 32 | 16 | 2:1 | + gqa_swa_hybrid — the SWA interleave exemplar |
Adoption (of ~3,325 current fingerprints): the gqa tag is on 244, gqa_swa_hybrid on 76,
a bare sliding_window on only 13. Read those honestly: the gqa tag is applied conservatively
— many more models use GQA head configs without carrying the explicit tag (Llama-3.1-8B above is
32/8 GQA), so 244 is a floor, not the true prevalence. What the counts do show cleanly: SWA appears
almost always as a hybrid (76) rather than pure (13). Why hybrids dominate — quality vs kernel
convenience — is a question we don't infer from the counts.
The honest trade-off (the Verdict)
- Which cost (keep them separate): GQA buys cost #3 (KV-cache memory) by the factor
G(the query/KV-head ratio); it also trims K/V-projection FLOPs byGand shrinks those projection matrices slightly, but leaves the attention FLOPs and the overall parameter count essentially unchanged. SWA buys cost #2 (per-token attention compute, O(W)); it buys cost #3 only with cache eviction. - What it trades: modeling capacity. Fewer KV heads = fewer independent key/value subspaces; windowed layers can't see far on their own. GQA's quality cost is empirically small (why it's the default); MQA's is larger in practice; pure SWA's is enough that hybrids dominate — but each of those is a measured (Tier-1) statement, not something the config proves.
- The knobs are real and visible: the group ratio
G(4:1 → 8:1 above) and the global:local layer mix are exactly the levers a lab tunes, and you can read them straight off the fingerprint. - Bridge to Module 4: GQA/MQA shrink the cache by sharing or dropping heads; SWA by shortening the window. The next rung, MLA, shrinks it a third way — by compressing K/V into a latent — and beats all of these on cache size.
- Tier discipline: head counts and window sizes are Tier-2 (config). Whether a specific GQA ratio or SWA window hurts a given model's quality is Tier-1 (measured) — not something we read off the config, and not something we imply from it.
Glossary delta
MHA / GQA / MQA · KV head vs query head · group ratio (G) · sliding-window attention
(SWA) · local vs global layers · gqa_swa_hybrid · fused_qkv (a compute nicety, not a
cache change)
Prev: Module 2 — Position · Next: Module 4 — Latent attention (MLA) Evidence: atlas snapshot 2026-08-13. Exemplars: Llama-3.1-8B, Qwen3.6-27B, gpt-oss-120b, gemma-4-31B-it.