A living post. The diffusion timeline and the US-adoption findings below are settled. The nineteen-to-five ratio is not, for a reason we explain in full. We will update this page rather than repost, and we will say what changed.
Last July we counted MLA adoption and found it concentrated almost entirely in Chinese labs. A fair reader asked the obvious follow-up: is that history, or is it still happening? We now have a technique we can watch diffuse in real time.
347 days, six labs
DeepSeek described Sparse Attention — DSA — in the DeepSeek-V3.2-Exp technical report, released 29 September 2025. It replaces full attention with a lightweight indexer that scores past tokens and runs full attention only on the top-k, turning the quadratic cost of long context into something close to linear.
Here is every lab that has shipped it in a production model since, dated by first release. Every one of them is Chinese:
| Lab | First model | Days after origin | Models |
|---|---|---|---|
| DeepSeek | DeepSeek-V3.2-Exp | origin | 13 |
| Zhipu AI | GLM-5 | 135 | 10 |
| MindLab | Macaron-V1-Preview | 249 | 4 |
| Meituan | LongCat-2.0 | 273 | 2 |
| openPangu | openPangu-2.0-Flash | 275 | 2 |
| Shanghai AI Lab | Atria-Dawn-Preview | 347 | 3 |
Forty-nine models in total. Every originating lab is Chinese. No US or European lab has designed a model around it.
That last sentence needs a caveat, because NVIDIA appears in our DSA data thirteen times. Every one of those is an NVFP4 re-quantization of a Chinese checkpoint — GLM-5, DeepSeek-V3.2, V4-Pro, V4-Flash. That is deployment engineering, not architecture adoption: the design decision was made in China, and NVIDIA is porting the result to its own number format. It matters because counting publishers rather than designers would show US participation in this technique that does not exist.
So we counted the whole vocabulary
DSA is one technique. We track thirty-one that materially affect efficiency — how much compute a token costs, how many bytes move, how much memory a model needs. We tagged each with the lab that first described it in public.
Nineteen were first described by Chinese labs. Five by American ones.
The five American ones are Mixture of Experts (Google — Shazeer et al. 2017 at LLM scale, popularised for dense transformers by Switch Transformer in 2021), partial rotary embeddings (EleutherAI, 2022), Grouped Query Attention (Google, 2023), YaRN context scaling (Nous Research, 2023) and Microscaling FP4 (Microsoft and NVIDIA, 2023).
Three caveats a skeptic should reach for immediately, and we would rather raise them ourselves.
First, origin here means the first lab to describe a technique in public, which is not the same as who had the idea. Corporate research that never gets published is invisible to this method, and at least one case is genuinely convergent: partial rotary embeddings were EleutherAI’s in 2022 and were independently rediscovered by both DeepSeek and StepFun.
Second, the count weighs a paradigm like Mixture of Experts exactly the same as a narrow change like sigmoid routing. Nineteen-to-five counts entries, not importance. The two most-used techniques in our entire atlas are both American: MoE appears in 321 models and GQA in 308. Nothing gets built without them.
Third — and this one came from our own review process, not from a reader — the denominator is incomplete. Our thirty-one are the techniques we have formally categorised. Thirteen more sit in the atlas uncategorised, and one of them is FlashAttention, which is American, originated by Tri Dao at Stanford in 2022, and arguably the single most consequential attention-efficiency technique in production anywhere. Its absence is a filter artifact rather than a judgement, but the effect is the same: a set that omits FlashAttention while counting sigmoid routing is not a set anyone should accept at face value. Several Chinese techniques are missing for the same reason, so we do not expect the ratio to move much — but we are not going to ask you to take that on trust before we have done the work. We are redoing the categorisation and will update this post with the corrected figure.
So the defensible version of the claim is narrower than the raw ratio, and it is about rate rather than history.
No US-origin technique in our categorised set postdates 2023 — though FlashAttention-3 landed in 2024, and whether that counts as a new technique or a new version of one is a fair argument to have. The Chinese ones are overwhelmingly 2024 and later, and nine of the nineteen come from DeepSeek alone.
The part we did not expect
Having established that origination has moved, the natural next question is whether anyone in the US is building with the results. We assumed the answer would be thin.
It is not. At least forty-five models from six American labs already carry Chinese-origin techniques in architectures they designed themselves — NVIDIA, Allen AI, Microsoft, OpenAI, Poolside and Thinking Machines. Most often multi-token prediction and shared experts, both DeepSeek’s. Read that as a floor rather than a total, for a reason the next paragraph makes uncomfortably clear.
The clearest case is Thinking Machines’ Inkling. It is a 944-billion-parameter omni-modal model on a novel architecture, built from scratch by an American lab. It runs DeepSeek’s multi-token prediction at a depth of eight, against DeepSeek-V3’s own one and the three-layer variants from Xiaomi, StepFun and Meituan.
The deepest MTP implementation in our atlas is American, and it is built on a Chinese technique.
We found that one by accident. Inkling’s config keeps its MTP settings in a nested block our detector did not read, so for months we had the model tagged as carrying neither MTP nor shared experts. A novel architecture with unfamiliar config keys reads as feature-poor to an automated scan, which systematically understates exactly the frontier models you most want to understand. We have fixed that rule. We assume there are others, which is why the forty-five is a floor.
Three budgets, not one speedup
Efficiency techniques get ranked in a single list, which hides the fact that they spend different currencies. A technique that halves the weights does nothing for the KV cache. Sorting them by the resource they economise makes a modern model’s stack legible.
Compute per token is FLOPs. It dominates prefill, where the whole prompt is processed in parallel, and it sets training cost. Mixture of Experts, DSA, zero/identity experts and variable experts per layer all spend here.
Bytes moved per token is memory bandwidth. It dominates decode, where every single token must stream the active weights and the entire KV cache. MLA, GQA, FP8, FP4 and hybrid linear attention spend here.
Tokens per forward pass is amortisation rather than reduction. Multi-token prediction and n-gram prediction make no token cheaper; they get more tokens out of one pass, so the payoff rides entirely on how often the draft is accepted.
Worked on DeepSeek-V3-0324, using the configuration values in our atlas. Note that gains within a budget multiply and gains across budgets do not — they are different pools, and adding them together is the most common way these numbers get overstated.
| Budget | What compounds | Result |
|---|---|---|
| Compute | MoE: 671.0 B total → 38.0 B active per token (256 experts, top-8) | 17.7× |
| Weight bytes | MoE 17.7× × FP8 2× — both act on the same pool | 35× |
| KV cache | MLA: 576 cached dims per layer per token, against 32,768 for equivalent MHA | 57× |
| Attention | DSA: O(L²) → O(L·k), added in V3.2 | → linear |
This is why no single technique is the efficiency win. A dense BF16 model of the same nominal capacity would read 35× more weight data and hold 57× more KV per token. Neither number would exist without the other being addressed: MoE alone leaves the KV cache untouched, and MLA alone leaves 671 B of weights to stream. The compounding is what made frontier-scale open weights runnable on commodity clusters.
Two honest limits on that table. These are ratios computed from the architecture, not measured throughput — we do not independently benchmark these models. And they are gross figures: routing overhead, expert-parallel communication and any quality cost are not netted out. DeepSeek’s own published claim for MLA is the more conservative “about 6% of equivalent MHA”; our 57× assumes an MHA baseline at the model’s own 128 heads and 128 head dimensions.
Why the framing matters
It is tempting to read nineteen-to-five as a scoreboard. We do not think that is the useful reading, for two reasons.
The American techniques are foundational in a way the count cannot express. MoE and GQA are the substrate everything else is built on. A list that puts sigmoid routing beside Mixture of Experts is measuring publication events, not significance.
And the flow is not one-directional. Origination has moved to Chinese labs; adoption has moved in both directions. American frontier labs are not ignoring this work — they are shipping it, in their own architectures, sometimes more aggressively than the originators.
The honest summary is that the United States has become an importer of this vocabulary rather than a bystander to it. Whether that is reassuring or alarming depends on what you think importing means.
The techniques, in full
Every technique named above, with its full name and where it was first described. Origin means the first public description, not necessarily who had the idea — see the first caveat. An em dash means contested, shared between labs, or not attributable to one country.
| Name | Full name | Origin | What it does |
|---|---|---|---|
moe | Mixture of Experts | US | Routed sparse activation across many specialised feed-forward networks; only a few run per token. |
gqa | Grouped Query Attention | US | Several query heads share a single key-value head, cutting KV cache proportionally. |
partial_rotary | Partial Rotary Embeddings | US | Rotary position encoding on a fraction of head dimensions; the rest carry position-invariant content. |
yarn_scaling | YaRN Context Scaling | US | Extends usable context by rescaling rotary position embeddings. |
mxfp4_native | Microscaling FP4 | US | Four-bit weight format with per-block scaling. |
mla | Multi-head Latent Attention | CN | Compresses keys and values into a low-rank latent space, keeping a small RoPE-carrying part uncompressed. |
dsa | DeepSeek Sparse Attention | CN | A lightweight indexer scores past tokens; full attention runs only on the top-k. Turns O(L²) into O(L·k). |
mtp | Multi-Token Prediction | CN | Dedicated layers predict one to three tokens beyond the usual target, used at inference for speculative decoding. |
shared_expert | Shared Expert | CN | An always-active expert carrying common knowledge, separate from the specialised routed ones. |
expert_bias | Expert Bias | CN | A learned per-expert bias corrects capacity utilisation, replacing an auxiliary balancing loss. |
noaux_tc | No-Auxiliary-Loss Top-k Correction | CN | Load balancing via expert-bias updates instead of an auxiliary loss term. |
sigmoid_routing | Sigmoid Expert Routing | CN | Sigmoid-plus-bias gating for expert selection, in place of softmax. |
group_routing | Group-Limited Expert Routing | CN | Top-k selection constrained to a subset of expert groups, cutting all-to-all communication across devices. |
fp8_native | FP8 Native Weights | CN | Weights distributed in FP8, with selective BF16 for sensitive layers. |
hybrid_linear_attention | Hybrid Linear Attention | CN | Mixes full or latent attention with linear-attention layers whose state stays near-constant regardless of context length. |
zero_identity_experts | Zero / Identity Experts | CN | Half the expert slots are pass-throughs that compute nothing, giving each token a dynamic compute depth. |
ngram_prediction | N-gram Prediction Layers | CN | An alternative to MTP as the draft source for speculative decoding. |
ngram_embedding | N-gram Embedding | CN | A very large lookup table of n-gram embeddings at an early layer — capacity without FLOPs. DeepSeek ships it as Engram. |
asymmetric_qv_dims | Asymmetric Q/V Dimensions | CN | Value projections narrower than query projections, saving both weights and KV. |
attention_sink_bias_swa | Attention Sink Bias | CN | A learnable bias at sink positions in windowed layers, allowing more aggressive KV reduction. |
attention_value_scale | Attention Value Scale | CN | A per-head learnable scale on attention output values. |
layer_specific_rope_theta | Layer-Specific RoPE Theta | CN | Different rotary base frequencies for global versus local attention layers. |
variable_experts_per_layer | Variable Experts Per Layer | CN | Layers carry different expert counts after pruning a uniform model (Layer-Adaptive Expert Pruning). |
masked_diffusion_lm | Masked Diffusion Language Model | CN | Begins from a fully masked sequence and iteratively denoises, rather than generating left to right. |
mopd | Multi-Teacher On-Policy Distillation | CN | Routes each prompt to a specialist teacher, with token-level log-ratio reward. |
gqa_swa_hybrid | GQA + Sliding-Window Hybrid | — | Alternates full-attention and windowed layers; windowed ones cap KV at the window size. Mistral and Gemma. |
swiglu_clamping | SwiGLU Activation Clamping | — | Bounds activation outliers in terminal layers. Claimed by both StepFun and OpenAI; precedence unresolved. |
use_qk_norm | Query-Key Normalisation | — | L2 or RMS normalisation of queries and keys before attention. |
norm_topk_prob | Top-k Probability Renormalisation | — | Renormalises routing probabilities after top-k selection. |
reasoning_effort_toggle | Reasoning Effort Toggle | — | Per-request reasoning depth on a single set of weights. |
flash_attention | FlashAttention | US | IO-aware exact attention that avoids materialising the attention matrix. Tri Dao, 2022. Uncategorised in our atlas — see the third caveat above. |
A note against our own record while we are here: our atlas describes dsaas “Dynamic Sparse Attention”, while DeepSeek’s V3.2-Exp report names it DeepSeek Sparse Attention. The lab’s name is the one used here, and we are correcting ours.
How to check this yourself
Every number here comes from public files. Adoption counts are configs on Hugging Face. Origination comes from the source paper each technique cites. Release dates are the publication dates of the models themselves. Nothing in this post requires access to anything we hold privately, and we would rather you verified it than took our word.
Two limits worth stating. We do not independently measure throughput — the efficiency claims attached to these techniques are the labs’ own, and a technique’s presence in a config tells you it was used, not how much it helped. And our organization table leaves 1,192 publishers without a country attribution, so both the Chinese and American lab counts should be read as floors rather than totals.