Parameter sparsity: Mixture-of-Experts

Axis: sparsity (orthogonal to the attention spine, Modules 3–6) · Prereq: Module 1 (the FFN is 70% of a dense model's weights) · Next: Module 8 (MTP) Hook: Attacks cost #1→#2 — turns a huge parameter count into a small FLOPs-per-token by using only a few of many expert FFNs per token. The technique that lets a 129 B model do the per-token math of a 3 B one — and the one most often misread.

Under the hood

An MoE layer replaces the dense FFN with:

• E expert FFNs (each a normal SwiGLU FFN, but narrower). Atlas range: 256–512 experts common.
• A router (a small linear layer) that scores the experts for each token and selects the top-k (typically k = 6–10). Only those k experts compute; the rest are skipped for this token.
• Optionally a shared expert — always active, carries the common/base computation so the routed experts can specialize. 117 of our MoE models carry one (shared_expert).
• Combine: the token's output = shared-expert output + weighted sum of its k routed experts.

So active params = the always-on core (embeddings/attention/norms/shared expert) + the k routed experts + the router, while total params ≈ all E experts. That gap is the Sparsity Ratio (SR) = total ÷ active. Two honesty notes baked in: the shared expert is often a different width than a routed one, and the router is a real hidden × E matrix — so a fingerprint-level active count (and thus SR) is an approximation, not an exact FLOPs statement.

Two hard problems MoE has to solve, visible in the fingerprint:

1. Load balancing. If the router sends everything to a few favorite experts, the rest are dead weight (routing collapse). Classic fix: an auxiliary loss that punishes imbalance. The modern fix (DeepSeek line) is auxiliary-loss-free balancing via a per-expert bias the router adds (noaux_tc, expert_bias) — balance without the aux-loss's quality tax. 53 models use noaux_tc.
2. Router sharpness. Softmax-over-experts vs sigmoid routing (score each expert independently) changes how decisively tokens are assigned. 62 models use sigmoid_routing.

Implementation trade-off (why not just crank E to the moon?): every expert's weights must live in GPU HBM, ready to be fetched the instant the router picks it. So E is bounded by VRAM to hold the model (cost #1), even though only k do math (cost #2). MoE trades disk/VRAM for compute — it does not make the model small.

Fingerprint evidence

(auto-populated — see _evidence/module-07.evidence.md; regenerate with db/gen_learning_evidence.py)

⚠️ Read the structure columns, not the SR column, to compare these models. SR (rightmost, in parentheses) is a parameter-usage ratio only — not latency, not cost, not a ranking. It's Tier-2 structure with no measured speed claim. What actually characterizes a MoE is experts + top-k (how many of how many fire per token) and the active count. SR is derived from those; don't rank models by it.

(SR is computed from the atlas's exact total/active then rounded, so Total ÷ Active of the independently-rounded cells won't reproduce it to the last decimal.)

The teaching centerpiece — same label, very different sparsity. DeepSeek-V4-Pro routes to 6 of 384 experts while Nemotron-3-Ultra routes to 22 of 512 — Nemotron uses ~3.6× more of itself per token, and their SRs differ 15× (60× vs 4×). The honest lesson: "it's an MoE" tells you almost nothing until you read top-k and the active count. One caveat we state rather than hide: this is not a controlled comparison — the two models also differ in total size and expert count, so the SR gap is driven by top-k and scale together. The clean, isolated lever is top-k (experts used per token); the pair above just shows how far apart two "512-ish-expert MoEs" can land. Ling-3.0-flash shows the other axis: a modest 129 B total but a 2.8 B active footprint — frontier-style sparsity at a size you can actually serve.

The honest trade-off (the Verdict)

• SR is NOT a speed or cost number (guardrail, load-bearing here). Ling's 45× SR does not make it 45× cheaper to serve: you still need VRAM to hold all 129 B, and MoE inference is frequently memory-bandwidth bound — the expert weights must be streamed from HBM even though only 8 run. A high-SR model with poorly-localized routing can be slower than a low-SR one. Read SR with a measured decode latency / tokens-sec, never alone. (Where we have that measurement we cite it; for several of these we have structure only — Tier-2 — and say so.)
• What MoE buys and what it costs: buys cost #2 (FLOPs-per-token ↓, big); costs #1 (parameter count ↑) and pressures #5 (VRAM footprint = the full total, plus routing overhead).
• Co-occurrence, not causation. These models pair MoE with MLA, MTP, DSA, etc. We are not claiming "MoE caused metric X" — we have no per-technique ablation. Module 11 decomposes the full stacks and is explicit about that limit.
• What we have measured vs not: we have behavioral results (ARS, worldview, claims) on several of these exact models (e.g. Ling, Hy3) — those are Tier-1, dated. Per-expert routing quality and serving-latency-vs-SR are not yet in the atlas; that's honest future work, not a current claim.

Glossary delta

expert / expert FFN · router / gating · top-k routing · shared expert · routing collapse · auxiliary-loss-free balancing (noaux_tc / expert bias) · sigmoid vs softmax routing · memory-bandwidth bound

Prev: Module 6 — Frontier attention sparsity (DSA) · Next: Module 8 — Decoding acceleration (MTP) Evidence generated from atlas snapshot 2026-08-13.