Axis: decoding (orthogonal to the attention spine and to ) · Prereq: Module 1 (autoregressive generation) · Next: Module 9 (Precision) Hook: Every technique so far shrank memory or compute per token. attacks the fourth cost — decode latency — the fact that a model normally produces exactly one token per forward pass, serially. MTP lets it propose several and verify them at once. Uniquely, it's lossless.
The 2-minute version (no math)
Normally a model writes one token, then runs the whole network again to write the next — strictly one-at-a-time, which is why generation feels sequential. MTP adds a small extra "head" that guesses the next couple of tokens ahead of time. A single fast pass then checks those guesses; the ones that match what the model would have produced anyway are kept, the rest discarded.
The output is identical to normal decoding — MTP doesn't change what the model says, only how many expensive passes it takes to say it. So unlike quantization or , there's no quality trade — it's a pure speed win, as long as the guesses are often right. It's the model speculating about itself.
Under the hood
- Training: add one or more extra prediction heads (
num_mtp_layers) that predict token t+2 (and sometimes further) alongside the main head's t+1. This also gives denser training signal (each position supervises multiple future tokens), a quality-side bonus separate from speed. - Inference — self-speculative decoding: the MTP head cheaply drafts the next k tokens; the full model then verifies all k in a single parallel forward pass (the drafts are appended and scored together under the usual causal mask, so position t+2 is evaluated alongside t+1 — no extra serial passes). Accept the longest correct prefix, resample the first rejected token, repeat.
- What "lossless" means, precisely: the output distribution is unchanged — not that compute is saved unconditionally. The standard speculative-sampling accept/resample rule preserves the target model's distribution at any temperature (not just greedy): a drafted token is accepted with a probability that corrects for the gap between the draft head and the true model, and rejections resample from the true . So the text you get is exactly what plain decoding would produce; MTP only changes how many passes it took.
- Why it's "self"-speculative: classic speculative decoding needs a separate small draft model; MTP folds the drafter into the model itself (the extra head), so there's no second model to serve.
Fingerprint evidence
(auto-populated — _evidence/module-08.evidence.md)
| Model | num_mtp_layers | Note |
|---|---|---|
DeepSeek-V4-Pro-0813 | 1 | one extra head (the common setting) |
NVIDIA-Nemotron-3-Ultra-550B | 1 | " |
Qwen3.8-2.4T-A95B | 1 | " |
Across the atlas: 138 current models carry MTP; 132 use a single extra head (num_mtp_layers=1)
and only 6 use 3. So MTP in practice is "one speculative head," not a deep stack — a small, cheap
addition. The headline: MTP has gone from a DeepSeek novelty to broad adoption across labs.
The honest trade-off (the Verdict)
- Which cost: buys cost #4 (decode latency) with no quality trade (lossless output, per above) — unique among the guide's techniques. Its price is training complexity and a tiny number of extra head parameters.
- When it backfires: the win depends on the acceptance rate, which collapses on high-entropy outputs (hard reasoning, novel code) where the draft head guesses wrong. Then you pay the extra head and the verification pass for near-zero speedup — MTP can be a net loss. So a serving stack should gate it (enable when acceptance is high, skip when it isn't). Present ≠ always-worth-it.
- The speedup is measured, not structural (guardrail): actual acceleration depends on the acceptance rate — how often the draft is right — which varies by workload and is a measurement. The config tells you MTP is present (and how many heads), never the realized speedup.
- Present ≠ enabled (a real serving gotcha): MTP weights in the checkpoint don't mean MTP is active — the serving stack has to support self-speculative decoding to cash in the latency win.
- We have measured MTP as roughly lossless with a real speed gain on at least one team model (Nemotron-3.5) — cited, dated, not generalized.
Glossary delta
MTP (multi-token prediction) · num_mtp_layers · speculative decoding · self-speculative
decoding · draft / verify · acceptance rate · lossless acceleration
Prev: Module 7 — Mixture-of-Experts · Next: Module 9 — Precision Evidence: atlas snapshot 2026-08-13. Exemplars: DeepSeek-V4-Pro, Nemotron-3-Ultra, Qwen3.8.