Glossary

Every term the guide clicks through to. Definitions are plain-language but precise — the same ones that pop up when you hover a term in a module.

active parameters: The parameters actually used for a given token: the always-on core (embeddings, attention, norms, any shared expert) + the k routed experts + the router. In a dense model, active = total.
attention sparsity: Skipping most token-PAIRS in attention — computing attention over only a selected subset of past tokens (e.g. DSA, Module 6). Distinct from parameter sparsity (MoE). 'Sparsity' unqualified is ambiguous; this is the attention kind.
auxiliary-loss-free balancing: The modern DeepSeek-line way to keep MoE experts evenly used: add a per-expert bias to the router's scores instead of an auxiliary load-balancing loss — balance without the quality tax.
base architecture: The root pretraining architecture a model derives from — the structural fingerprint the detector matches a candidate against (e.g. deepseek_v3, qwen3, llama3, gemma, glm4). It is the lineage root. Distinct from a 'base model' (a pre-post-training checkpoint, before instruct/RLHF) and from a model's 'publisher' (whoever uploaded it). ModelDNA's detector currently recognizes ~40 base architectures.
d_model: The width of the residual stream — the single most important size number in a model; most other dimensions are derived from it.
DSA: DeepSeek Sparse Attention: a frontier attention-sparsity technique that skips most token-pairs to keep attention sub-quadratic while preserving quality. Distinct from MoE's parameter sparsity.
expert: One of the many small FFN sub-networks in a MoE layer. A router selects the top-k experts per token; only those compute (plus any always-on shared expert).
FFN: Feed-forward network: the per-token 'thinking' sub-layer of a transformer block, applied to each token independently. In modern models it uses the SwiGLU variant and holds the majority of the weights.
FLOPs-per-token: The arithmetic done to generate one token. In a dense model it scales with total parameters; MoE cuts it by using only the active parameters.
FP8-native: Training or running a model in 8-bit floating point from the start (a training-time precision choice), rather than quantizing after the fact — different error profile from post-hoc 4-bit.
fused_qkv: Packing the Query, Key, and Value projections into one matrix multiply — a compute/kernel nicety. It does NOT change the KV-cache.
GQA: Grouped-query attention: several query heads share one Key/Value head, so the KV-cache shrinks by the group ratio G (e.g. 4:1). Near-universal because the quality cost is small.
head_dim: The width of a single attention head. KV-cache size per layer is proportional to (number of KV heads × head_dim × 2).
kv_lora_rank: The dimension of MLA's cached latent — the size of the compressed Key/Value summary stored per token (e.g. 512). Independent of head count.
KV-cache: The cached Keys and Values of every past token, kept during generation so each new token can attend to the history. It grows with context length and is usually the real wall for long context — not parameter count. Queries are never cached.
MHA: Multi-head attention: the original attention design where every query head has its own Key and Value heads. Maximum quality, maximum KV-cache. Modern models shrink it with GQA.
MLA: Multi-head latent attention (DeepSeek line): compress each token's Keys/Values into one small latent vector (kv_lora_rank, e.g. 512), cache only that, and reconstruct per-head K/V on the fly. Shrinks the KV-cache ~28× vs MHA — a memory-for-compute trade.
MoE: Mixture-of-Experts: replace the one big FFN with many smaller expert FFNs plus a router that sends each token to only a few. The model holds huge total parameters while using few active ones per token — parameter sparsity.
MQA: Multi-query attention: the extreme of GQA where all query heads share a single Key/Value head. Smallest KV-cache, but a larger quality cost, so it's less common than GQA.
MTP: Multi-token prediction: extra heads that let a model propose more than one token per step, used for self-speculative decoding — a decode-latency win.
parameter sparsity: Using only a fraction of the WEIGHTS per token — a router sends each token to a few expert FFNs (MoE, Module 7). Distinct from attention sparsity (DSA). 'Sparsity' unqualified is ambiguous; this is the parameter kind.
partial rotary: Applying RoPE to only a fraction of each head's dimensions, leaving the rest position-free. Cheaper, but the unrotated dimensions reintroduce some absolute-position sensitivity.
pre-norm: Placing the normalization before each sub-layer (rather than after). It's what makes very deep transformer stacks trainable, and is now universal.
residual stream: The running sum every sub-layer reads from and writes to — the 'highway' that lets information and gradients skip the full depth of the network. Its width is d_model.
RMSNorm: Root-mean-square normalization: a cheaper normalization than LayerNorm (rescale by RMS, no mean-subtraction, one learned gain). Placed before each sub-layer (pre-norm), it keeps deep stacks trainable and is near-universal.
RoPE: Rotary position embedding: encode word order by rotating each query/key (in 2-D pairs of dimensions) by an angle proportional to position. Because attention is a dot product, this makes scores depend on relative distance — for fully-rotated heads.
rope_theta: The base hyper-parameter that sets the spread of RoPE's rotation frequencies. Increasing it is one lever for extending context length.
router: The small linear layer in a MoE that scores the experts for each token and selects the top-k. Poor routing that favors a few experts is 'routing collapse'.
shared expert: An expert that is always active for every token, carrying the common computation so the routed experts can specialize. Because it's always on, MoE is not simply 'only some experts wake up'.
Sparsity Ratio: Total parameters ÷ active parameters — how sparse a model's parameter usage is (e.g. 60×). A parameter-usage headline, NOT a speed or cost ratio: it ignores memory bandwidth and is undefined for dense models.
SSM: State-space model (e.g. Mamba): an O(n) sequence-mixing alternative to softmax attention, often interleaved with attention layers in a hybrid stack.
SWA: Sliding-window attention: a layer attends only to the last W tokens. It reliably cuts attention compute to O(W); it cuts KV-cache memory only if the serving stack evicts tokens outside the window. Usually deployed as a hybrid with a few full-attention 'global' layers.
SwiGLU: The modern feed-forward (FFN) variant: three matrices (gate, up, down) with a SiLU-gated product — down(SiLU(gate·x) ⊙ (up·x)). It's where most of a dense model's parameters live (~70%), which is why MoE targets the FFN.
Tier-1 / Tier-2: ModelDNA's evidence tiers. Tier-2 = config/structure-derived facts — what a model IS, read from its config and weights (most of this guide). Tier-1 = MEASURED behavior — what a model DOES, from actually running it (agentic, worldview, safety tests), always dated. The discipline: a Tier-2 structural fact (e.g. a Sparsity Ratio) is never presented as a Tier-1 performance claim (e.g. speed).
top-k routing: The number of experts (k) a MoE router activates per token — the key lever for how sparse a model is. Low k (e.g. 6) = very sparse; higher k (e.g. 22) = much denser.
weight forensics: Reading the weight tensors directly (not just the config) to answer what the config can't: a per-tensor diff against a known base (is this really its own model, or a renamed copy?), or detecting modifications like abliteration. The deepest structural read — still 'structure,' just at the weight layer rather than the config.
YaRN: A context-extension scheme: NTK-aware interpolation that stretches low RoPE frequencies while preserving high ones, blended with a smooth ramp and an attention-temperature correction. Lets a model reach far longer contexts than it trained on.