Axis: — (foundation) · Prereq: Module 1 (attention) · Next: Module 3 (the wall) Hook: Attention is order-blind — it treats the input as a bag of tokens. Position has to be injected. And "context length" is really a positional problem: the reason a model can (or can't) read a 1M-token document is entirely about how it encodes position.
The 2-minute version (no math)
Attention, on its own, doesn't know that "dog bites man" differs from "man bites dog" — it sees the same three words. So every model has to tell attention where each token sits. Modern models do this elegantly: they rotate each token's query and key vectors by an angle that depends on its position (this is , rotary position embedding). Two tokens close together end up at similar angles; far apart, very different — so attention can feel relative distance for free.
The catch: a model only learned the angles up to the length it trained on (say 128K tokens). Ask it to read something longer and the angles run off the edge of what it's seen, and quality falls apart. So there are tricks — , NTK, longrope — that stretch or re-space the rotations to extend a model's configured window (some to 1M tokens). They extend the configured window, but not necessarily the effective context — the range where quality actually holds. That gap (configured vs effective) is the honest catch this module ends on.
Under the hood
The lineage, briefly, because it explains why RoPE won:
- Absolute position embeddings (add a per-position vector) — simple, but don't generalize past the trained length and encode position absolutely, not relatively.
- Rotary (RoPE) — the modern default. Instead of adding anything, take each query and key as a set
of 2-D pairs of dimensions and rotate each pair by an angle
position × θ_i, where the frequenciesθ_irun from high (changing fast across positions) to low (changing slowly) acrosshead_dim. Because the attention score is a dot product of Q and K, this rotation makes the score depend only on the relative offset between two tokens — for a fully-rotated head. (Partial RoPE, below, leaves some dimensions unrotated, which reintroduces mild absolute-position sensitivity in that subspace — a real caveat, not a rounding detail.) A single base hyper-parameter,rope_theta, sets the frequency spread.
Extending context = manipulating those rotations. A model trained to 128K can be pushed further by rescaling the frequencies so the never-before-seen long positions land inside the range the model already understands:
- linear / NTK scaling — divide the positions (crude but cheap).
- YaRN — NTK-aware interpolation: interpolate the low frequencies while largely preserving the
high ones, blended with a smooth ramp (not a hard cut, via
beta_fast/beta_slow) plus an attention-temperature correction that keeps logits well-scaled. The current workhorse for big extensions; DeepSeek-V4-Pro configures it with a 1,048,576-tokenmax_position_embeddings. - longrope / dynamic — variants that adjust per-length.
- (
partial_rotary) — apply the rotation to only a fraction of each head's dimensions and leave the rest position-free; Qwen3.8 does this (per our fingerprint).
Position is free of learned weights (RoPE adds none of its own; the scaling schemes add a few
hyper-parameters to tune — rope_theta, the YaRN betas — not parameters) — but it is the gatekeeper
of cost #3: a longer context is a bigger KV-cache, which is the wall Module 3 opens.
Fingerprint evidence
(auto-populated — _evidence/module-02.evidence.md)
| Model | rope_type | Max context | Note |
|---|---|---|---|
Llama-3.1-8B | llama3 | 131,072 | NTK-style scaling baked into the base |
gpt-oss-120b | yarn | 131,072 | YaRN scaling (no partial rotary in our fingerprint) |
GLM-5 | default | 202,752 | scaled RoPE, no exotic type tag |
Qwen3.8-2.4T-A95B | (partial) | 262,144 | partial-rotary hybrid |
DeepSeek-V4-Pro-0813 | yarn | 1,048,576 | YaRN to a 1M-token window |
Across the whole atlas, default RoPE dominates (1,969 current models), with 127 on YaRN, 89
on the Llama-3 scaling, 56 on longrope, 46 dynamic — i.e. plain RoPE is universal and scaling is the
active frontier where labs differentiate on long-context reach.
The honest trade-off (the Verdict)
max_position_embeddingsis a claim of reach, not a guarantee of quality at that reach. A model advertising a 1M context can still degrade badly in the middle of a long document ("lost-in-the-middle"). The atlas records the configured window (, structure); whether quality holds at length is a measured (Tier-1) question we do not read off the config, and we don't imply we do.- Which cost it touches: position itself is ~free (no params, negligible compute). Its real consequence is indirect — enabling long context, which drives cost #3 (KV-cache). That hand-off is Module 3.
- Scaling is a trade, not a free lunch: stretching rotations to reach further generally costs some fidelity near the original training length; the "best" scaling is model- and length-specific.
Glossary delta
position embedding (absolute vs rotary) · RoPE · rope_theta · relative position ·
context length / max_position_embeddings · YaRN / NTK / longrope · partial rotary
Prev: Module 1 — The dense baseline · Next: Module 3 — Attention & the KV-cache memory wall Evidence: atlas snapshot 2026-08-13.