Axis: attention spine (sibling of Modules 3, 5, 6) · Prereq: Module 3 ( shares KV heads) · Next: Module 5 (linear attention & ) Hook: Attacks cost #3 ( memory) the hardest of any technique here. Module 3's GQA shared KV heads to shrink the cache; goes further — it stores a single small latent per token and reconstructs each head's keys and values on the fly.
The 2-minute version (no math)
Recall the wall from Module 1: to keep generating, a model must remember every past token's Keys and Values — the KV-cache — and it grows with context length until it eats all your GPU memory. Module 3's GQA helped by making groups of heads share one set of K/V.
MLA (Multi-head Latent Attention, from the DeepSeek line) takes a different, more aggressive route: instead of caching K and V for every head, it caches one small compressed summary ("latent") per token, and regenerates the per-head keys and values from that summary whenever they're needed. Think "store a zip file, unzip on demand" rather than "store every file." The cache shrinks by an order of magnitude — which is how these models serve very long contexts and large batches. The price is a bit more arithmetic (the unzip) and famously fiddly GPU kernels, which is why not everyone adopted it.
Under the hood
MLA restructures attention around a low-rank latent:
- Down-project each token's hidden vector into a small latent
cof dimensionkv_lora_rank(512 in GLM-5 and Kimi-K2.6). Onlyc(plus the small position key below) is cached — its size is independent of head count. - Up-project
cback into per-head Keys and Values when attention runs. Crucially, the up-projection matrices can be absorbed into the query and output projections, so the full per-head K/V are never persistently cached — they're materialized only transiently, in compute, for the current token's attention. - Decoupled . Rotary position (Module 2) can't be applied to a compressed latent cleanly, so
MLA carries a small separate position-bearing key — a shared rope key (typically 64 dims
total, not per head) — alongside the latent. The cached state is thus
kv_lora_rank + rope_dim(≈ 512 + 64 = 576) per token, per layer. - Query compression too. A
q_lora_rank(1,536 in Kimi, 2,048 in GLM-5) low-ranks the queries — a training/compute nicety, not a cache saving (queries aren't cached).
The payoff, as a cache-size comparison (per token, per layer — the number Module 1 taught you to compute), for a Kimi-scale model (64 heads × 128 dims):
| Scheme | Cached per token / layer | vs full |
|---|---|---|
| Full MHA | 2 × 64 × 128 = 16,384 values | 1× |
| GQA (8 KV heads) | 2 × 8 × 128 = 2,048 | ~8× smaller |
| MLA | kv_lora_rank 512 + rope 64 = 576 | ~28× smaller than MHA, smaller even than GQA |
(Counts are cached elements per token per layer; × 2 bytes for the BF16 footprint — so MLA ≈ 576 × 2 = 1,152 bytes/token/layer.)
That is the point: MLA shrinks the KV-cache ~28× vs MHA. It does not eliminate the wall — at a 1M-token context, 576 elements × 2 bytes × ~60–128 layers still runs to tens of GB — but it turns "infeasible on one node" into "comfortable across a few," which is exactly what long-context serving needs.
Fingerprint evidence
(auto-populated — _evidence/module-04.evidence.md)
| Model | Heads | kv_lora_rank (cached latent) | q_lora_rank |
|---|---|---|---|
GLM-5 | 64 | 512 | 2,048 |
Kimi-K2.6 | 64 | 512 | 1,536 |
DeepSeek-V4-Pro-0813 | 128 | (MLA; latent ranks not in the config-only record) | — |
MLA is present in 77 current atlas models — and note who: it is concentrated in the DeepSeek lineage and its adopters (GLM, Kimi), not spread evenly across all labs. That non-adoption is itself intelligence (see the Verdict).
The honest trade-off (the Verdict)
- What it buys / what it costs: buys cost #3 (KV-cache) enormously — enabling long context and big batch on fixed VRAM — at the price of more cost #2 (the up-projection arithmetic) and, in practice, serious kernel-engineering effort. It is a memory-for-compute trade, the mirror image of .
- Why MLA didn't spread despite being excellent — an explicitly UNCONFIRMED working hypothesis, motivated by the atlas distribution (a lens, not a finding — we have no causal evidence). MLA needs bespoke attention kernels and a specific up-cycling path, so labs with a mature GQA stack plausibly stay put even though MLA is the stronger KV compressor. We call this the switching-cost hypothesis: it reads the 77-vs-thousands adoption split as economics rather than a quality verdict — a lens the data motivates, which controlled evidence (not co-occurrence) would be needed to confirm.
- is irrelevant here. MLA is an attention technique; it changes cost #3, not the parameter sparsity. Don't let a model's MoE SR bleed into how you judge its attention.
- Tier discipline: latent ranks and head counts are (config). The realized long-context quality and the decode-latency cost of the up-projection are Tier-1 (measured) questions — we cite them where we have them and flag where we don't (DeepSeek's own record above is missing its ranks, and we show that rather than invent them).
Glossary delta
MLA (multi-head latent attention) · latent / low-rank projection · kv_lora_rank ·
q_lora_rank · weight absorption · decoupled RoPE · memory-for-compute trade
Prev: Module 3 — Attention & the KV-cache memory wall · Next: Module 5 — Beyond softmax: linear attention & SSM Evidence: atlas snapshot 2026-08-13. Exemplars: GLM-5, Kimi-K2.6 (latent ranks); DeepSeek = origin.