A modern 4B language model is trained on about 25 trillion tokens: at least 6 × 10²³ operations, around 420,000 H100 hours for the final run. Where the numbers come from, a simple modern design, and what smaller budgets buy.
A modern 4-billion-parameter language model is trained on roughly 25 trillion tokens of text. That takes at least 6 × 10²³ arithmetic operations: around 420,000 hours on NVIDIA H100 GPUs for the final training run alone, and more than twice that for the whole project. Here is where those numbers come from, what a simple modern design looks like, and what smaller budgets buy. Building from scratch costs far more than adapting an existing open model; the benefit that is hardest to get any other way is control over every token that went in.
At a glance (4B parameters, 25T tokens, final run):
- ~25T training tokens
- At least 6 × 10²³ operations
- ~420,000 H100 GPU-hours, about 17 days on 1,024 GPUs
- ~500–630 MWh of electricity, about a year’s use for 50–60 US homes
The data: far more than the textbook rule
The well-known “compute-optimal” rule from 2022 (Chinchilla) is about 20 training tokens per parameter, which would be 80 billion tokens for a 4B model. That rule answers a different question: the best model for a fixed training budget. Labs now train small models hundreds of times past it. The usual explanation is that a small model is cheap to run, so spending more on training to make it better pays off every time it is used. These are the counts the labs state on their own model cards:
| Model | Parameters | Training tokens | Tokens per parameter |
|---|---|---|---|
| Lucie-7B (OpenLLM-France) | 6.7B | 3T | ~450 |
| Granite-3B-Code (IBM, 2024) | 3.5B | 4T | ~1,150 |
| Apertus-8B (Swiss AI) | 8.1B | 15T | ~1,860 |
| LFM2-2.6B (Liquid AI) | 2.6B | 10T | ~3,900 |
| Nanbeige4-3B (Kanzhun, 2025) | 3.9B | 23T | ~5,900 |
| Nanbeige4.2-3B (Kanzhun, 2026) | 4.2B | 28T | ~6,700 |
| LFM2.5-1.2B (Liquid AI) | 1.2B | 28T | ~24,000 |
Parameter counts are from the ModelAtlas catalogue’s own checkpoint analysis. Token counts are the labs’ claims and cannot be checked from the weights; some include repeated or upsampled data and mid-training.
The processing: one formula
training operations ≈ 6 × parameters × tokens
For every token, each parameter does about 2 operations on the forward pass (making a prediction) and about 4 on the backward pass (working out how to adjust itself). This is a minimum: it leaves out recomputing activations to save memory, which many runs do, and the cost of scoring every word in the vocabulary at the output.
For 4 billion parameters and 25 trillion tokens: 6 × 4×10⁹ × 25×10¹² = 6 × 10²³ operations. An H100 peaks at about 989 trillion operations per second in BF16. Well-run training sustains roughly 30–45% of that; we use 40%. At 30%, every time, cost and energy figure below grows by a third.
| What it means in practice | 4B model on 25T tokens, final run |
|---|---|
| GPU time | ~420,000 H100-hours |
| Calendar time | ~17 days on 1,024 GPUs; ~5 weeks on 512 |
| Rented compute | ~$0.85–1.3 million at $2–3 per GPU-hour |
| Electricity | ~500–630 MWh, at 1.2–1.5 kW per GPU including servers and cooling |
| Memory | ~64 GB of weights, gradients and optimizer state (16 bytes per parameter), plus activations. Memory and communication shape how the job is split across GPUs, but the number of operations sets the budget. |
Reality check. Lucie-7B’s card states it was trained on 512 H100s for about 550,000 GPU-hours on 3T tokens. The formula gives a floor of about 85,000. The real run took 6.5 times the floor; restarts, experiments and lower efficiency on a shared supercomputer are the likely reasons. Treat the formula as the minimum, not the budget.
The whole project, not just the run
- Data preparation. 25T tokens is about 100 TB of text: web crawls, code, books and papers, filtered and deduplicated, and increasingly synthetic text written by larger models. Building it is a large compute job of its own.
- Experiments. Many small trial runs to choose the architecture and data mix, plus runs that fail.
- After pretraining. Mid-training (longer context, more math and code), instruction tuning, and reinforcement learning, which reasoning models spend heavily on.
A common rule of thumb is that the final run is a third to half of total compute. For a modern 4B model that puts the whole project at about 0.8–1.3 million H100-hours, or roughly $1.7–3.8 million of compute before staff and data costs, if everything runs at 40% efficiency. At Lucie-like efficiency it would be several times more.
A simple, modern design to start from
Small open models from Alibaba (Qwen3), Hugging Face (SmolLM3), IBM (Granite 4), Allen AI (OLMo 2) and Kanzhun (Nanbeige 4) have converged on nearly the same plain recipe; the ModelAtlas catalogue holds all of their configurations. Nothing in it is exotic, which is what you want for a first build: every part is supported by standard training and serving software. We show it at 1.7B parameters, the size we suggest starting with (see the budgets below).
| Choice | Setting (~1.7B parameters) | Why |
|---|---|---|
| Type | Dense decoder-only transformer | Simplest to train and serve. Mixture-of-experts pays off at larger scale. |
| Size | 28 layers, width 2,048 | Same shape as Qwen3-1.7B (1.72B in our count). |
| Attention | Grouped-query, 16 query and 8 key/value heads, head size 128 | Halves the memory each conversation needs when serving. Four of the five families use it; OLMo 2’s 1B does not. |
| Positions | RoPE; train at 4K context, extend to 32K later | Universal; long context is added cheaply in mid-training. |
| Feed-forward | SwiGLU, inner width 6,144 (3× width) | SwiGLU has been standard since Llama; all five use it. The inner width varies by model. |
| Normalization | RMSNorm before each block, plus QK-norm | QK-norm (Qwen3, OLMo 2) keeps large-learning-rate training stable. |
| Biases | None | None of the five use them. |
| Embeddings | Input and output tied | Saves ~0.3B parameters at this size (Qwen3, SmolLM3, Granite 4). |
| Tokenizer | Reuse an open ~130–150K vocabulary | Training a new one adds risk for little gain. Check its license. |
| Training | AdamW, BF16, warmup-stable-decay schedule | The stable phase can be extended or stopped without restarting. |
Counting the parameters. At 1.7B: embeddings 151,936 × 2,048 ≈ 0.31B; each layer has ~12.6M in attention and ~37.7M in the feed-forward block, so 28 layers ≈ 1.41B; total ≈ 1.72B. At 4B, Qwen3-4B uses the same recipe with 36 layers, width 2,560, 32 query heads and a wider feed-forward block (9,728, about 3.8× width): embeddings ≈ 0.39B, each layer ≈ 26.2M attention + 74.7M feed-forward, total ≈ 4.02B.
What a reasonable investment buys
Final-run compute from the formula, at 40% H100 efficiency and $2–3 per GPU-hour. Budget two to three times this for the whole project.
- Pilot: 0.6B on 100B tokens. 3.6 × 10²⁰ operations; ~250 H100-hours rented (~$500–750), or roughly two weeks on two workstation GPUs (estimate). Proves the data pipeline, training code and evaluation end to end. Not a useful model on its own.
- A sensible first build: 1.7B on 4T tokens. 4.1 × 10²² operations; ~29,000 H100-hours (~$60–85K), about 19 days on 64 H100s, ~35–45 MWh. A real, usable small model whose every training token is known. It will trail Qwen3-1.7B, which Qwen reports training on about 36T tokens.
- Stretch: 4B on 15T tokens. 3.6 × 10²³ operations; ~250,000 H100-hours (~$0.5–0.75M), about 20 days on 512 H100s, ~300–380 MWh. The same size as Nanbeige4-3B on about two-thirds of its data.
The trade-off. Continued pretraining of an existing open base model costs a small fraction of this, but the result inherits that model’s data and behavior. Building from scratch also lets you choose the tokenizer, the architecture and the license terms, but its main advantage is control over every token that went in, which matters when provenance is the point.
Sources: parameter counts and architecture settings from the ModelAtlas catalogue’s checkpoint and config analysis; training-token and GPU-hour figures from the labs’ own model cards (October 2026); Qwen3’s 36T figure is Qwen’s own report. Assumptions: H100 dense BF16 peak 989 TFLOPS at 40% sustained; $2–3 per H100-hour; 1.2–1.5 kW per GPU all-in; US home ~10.5 MWh per year; the workstation pilot assumes about 150 TFLOPS sustained per RTX 6000 Blackwell card, not measured. “Final run is a third to half of the project” is an industry rule of thumb, not our measurement.