The dense baseline (and its atoms)

Axis: foundation · Prereq: Module 0 (the five efficiencies) · Next: Module 2 (Position) Hook: The dense decoder pays all five costs in full. It's the reference every later technique is measured against — and understanding where its parameters and memory go tells you exactly what the rest of the guide is trying to shrink.

Under the hood

A dense decoder-only transformer is a stack of N identical blocks between an input embedding and an output projection:

tokens → [embed] → ( block ) × N → [final RMSNorm] → [unembed → logits] → next-token
                      │
     block = x → RMSNorm → Self-Attention → +x   (residual)
               → RMSNorm → FFN(SwiGLU)     → +x   (residual)

Note the pre-norm placement (RMSNorm before each sub-layer, not after): this is what makes deep stacks trainable, and it's now universal. The residual (+x) is the highway that lets gradients and information skip the whole depth.

The atoms (the panel moved these here — they're the building blocks, not an appendix):

• Self-attention (MHA — the conceptual form). For each token, project its hidden vector into a Query, Key, and Value (learned matrices W_Q, W_K, W_V). A token's Query is dotted against every prior token's Key to get attention weights, which mix the Values. Multi-Head = do this in h parallel subspaces (each of size head_dim) so different heads specialize (syntax, coreference, …), then concatenate and project out with W_O. In pure MHA every head has its own K and V; the very first efficiency (Module 3, GQA) is to share K/V across groups of heads — and our exemplar below already does this. Cost: O(sequence²) in compute, and — critically — you must cache every token's K and V to generate the next token. That KV-cache is the memory wall Modules 3–4 attack.
• FFN with SwiGLU. A two-layer MLP applied to each token independently, widened to an intermediate size (~3.5× d_model). Modern models use the SwiGLU variant: three matrices (gate, up, down) with a SiLU gate — down( SiLU(gate·x) ⊙ (up·x) ). This is where most of a dense model's parameters live (see below), which is why MoE (Module 7) targets the FFN.
• RMSNorm. A cheaper normalization than LayerNorm: rescale by the root-mean-square, no mean- subtraction, one learned gain vector. Ubiquitous (our atlas: silu_activation on 1,572 models, RMSNorm as the near-universal norm_class).
• Residual stream. The running sum every sub-layer reads from and writes to. Its width is d_model — the single most important number in the model.

Fingerprint evidence — meta-llama/Llama-3.1-8B (atlas record)

The canonical dense reference. Real fingerprint from our atlas:

Where the 8 billion parameters actually live (this is the load-bearing insight):

→ The FFN is 70% of the weights. That single fact explains the whole rest of the efficiency story: MoE (Module 7) makes the FFN sparse because that's where the parameters are; attention tricks (Modules 3–6) attack memory and compute, not parameter count, because attention is only 17% of the weights but 100% of the KV-cache growth.

What "baseline" means here (and what it doesn't): the baseline is the dense decoder — every token through every parameter, no MoE, no latent attention, no linear/SSM mixing. It is not a claim of "pure MHA": the real-world dense baseline already ships with GQA (our exemplar has 32 query heads but only 8 KV heads), because GQA is a near-free memory win everyone takes. We teach MHA as the conceptual atom — the un-shared form GQA is defined against in Module 3 — while the exemplar honestly shows the baseline already took step one down the attention spine. Nothing in our atlas is "unoptimized"; the baseline is architecturally dense, not historically pure.

The honest trade-off (the Verdict)

The dense baseline's "trade-off" is that it makes none — it pays every one of Module 0's five costs in full:

• Parameter count: every weight is stored and used.
• FLOPs-per-token: every token flows through all 8 B params → compute scales with total size.
• KV-cache memory: grows linearly with context length, and this — not parameter count — is what caps long-context on a given GPU. The load-bearing number, worked for Llama-3.1-8B in bf16: 32 layers × 8 KV heads × 128 dims × 2 (K and V) × 2 bytes = 131,072 bytes ≈ 128 KB per token (this assumes a bf16 KV-cache; an fp8 or int8 KV-cache — increasingly standard — roughly halves or quarters it, trading a little precision for reach). At a 128K-token context that's ~16 GB of KV-cache — on top of the 16 GB for the weights themselves. This is why the KV-cache, not the parameter count, is the long-context wall, and why Modules 3–4 (GQA, MLA) spend all their effort shrinking exactly this number.
• Decode latency: one forward pass per token, no shortcuts.

That's the point of a baseline: it's the honest, strong, expensive thing. Every subsequent module is a specific refusal to pay one of these bills — and names what it trades to do so.

What we have NOT measured here: this module is Tier-2 (config-derived structure). We are not claiming a quality number for "dense vs efficient" — that's a measured (Tier-1) question the later modules and the atlas answer per-model, dated.

Glossary delta

block / layer · residual stream · d_model · head / head_dim · KV-cache · pre-norm · RMSNorm · FFN / SwiGLU · MHA (multi-head attention) · logits / unembed

Prev: Module 0 — Orientation & the five efficiencies · Next: Module 2 — Position (RoPE & long context) Evidence generated from atlas snapshot 2026-08-13. Exemplar: Llama-3.1-8B.