Baidu published a frontier-scale open model before any mid- or post-training. Here is what specializing it would cost, who it would take, what the license allows, and where the real risk sits.
A team could turn Baidu’s ERNIE-5.5-Flash-Base into a long-context, coding and STEM model for about 200,000 H100-hours, roughly $400,000 to $690,000, and 12 to 21 people over four to six months. That is under a tenth of what the base cost to pre-train. The catch is not the compute. Baidu released weights and inference code but no training code, so the first two months go to building a trainer for an unusual architecture. And the license rules out the EU, the UK, South Korea and military use.
Why this checkpoint matters
Large open models almost always arrive already post-trained, with their maker’s choices about reasoning, tone and safety built in. ERNIE-5.5-Flash-Base, released on November 5, is one of very few frontier-scale checkpoints published before any mid- or post-training. Baidu says that is the point: “we let developers build on the model’s general capabilities and choose for themselves where to take it next.”
- It skips the most expensive step. Pre-training took at least 2.8×10²⁴ operations, about 2.5 to 3.5 million H100-hours. Few organizations can repeat that. Building on it is within reach of a well-funded team with the right engineers.
- The builder chooses the specialization. Mid-training on code, STEM and long documents, then post-training for agents or reasoning, decides what the model becomes.
- It is cheap to run for its size. Only 14B of its parameters are active per token, and most layers keep one cache entry per 128 tokens, so long contexts and reinforcement-learning rollouts cost far less than in a dense model.
- Its reported base scores are strong, but unverified. Baidu puts it ahead of DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base on 25 of 30 benchmarks, all from Baidu’s own unpublished evaluation pipeline, so the margins can’t be compared with other labs’ tables. We are running an independent check against DeepSeek-V4-Flash-Base under one neutral harness, with our predictions written down before the run, and will report the result.
The model in brief
The design borrows heavily from DeepSeek-V4 and adds several pieces of Baidu’s own. That combination is why training it needs custom work.
| Item | ERNIE-5.5-Flash-Base |
|---|---|
| Parameters | 286.2B by our count of the weight files (card: 280B); 14B active per token |
| Pre-training | 33.5T tokens from scratch, Muon optimizer, trained to 128K context |
| Context | 1M tokens configured for fine-tuning and inference |
| Attention | 37 Heavily Compressed Attention layers (one cache entry per 128 tokens) and 6 MLA layers with DeepSeek Sparse Attention |
| Experts | 512 routed (top 10) plus 1 shared per layer, in a half-width latent space, chosen by several routers |
| Other pieces | Hyper-connections (4 residual streams), gated attention, a final layer that doubles as the multi-token predictor, a MultiMax output head |
| Released | FP8 and BF16 weights (289 GB and 573 GB), inference through SGLang and a vLLM plugin |
| Not released | Training code, optimizer state, data, learning-rate schedule, any smaller sibling model |
Our count of the weight files comes to 286,245,622,106 parameters, 2% above the card’s 280B. The exclusions the card states don’t account for the gap, which may simply reflect how Baidu counts. It does not change anything below.
What the license allows
The ERNIE 5.5 Flash Base Model License permits fine-tuning, continued training and publishing derivatives, within limits that decide who a project like this can serve. Three consequences matter most:
- Data can flow in, not out. Other models’ outputs can train ERNIE where their licenses allow it. ERNIE’s outputs cannot train anything else.
- The market is the territory. A product with EU, UK or Korean users needs a different base.
- Hosting brings duties. Anyone offering it to others must keep abuse safeguards and a way to report violations, and must label published machine-generated material, code included.
The terms in full, as a plain-language reading, not legal advice:
| Use | Allowed? | Conditions |
|---|---|---|
| Research, education, evaluation | Yes, free | Keep the license and notices; mark modified files |
| Mid-training, fine-tuning, RL, merging, quantizing | Yes | The result is a “Derivative Model” under the same license |
| Commercial use, related revenue up to US$50M | Yes, free | One-time notice to Baidu; show “Powered by ERNIE” |
| Commercial use above US$50M | Only with Baidu’s prior written permission | Revenue counts whole products, all affiliates combined |
| Use, or use of outputs, in the EU, UK or South Korea | No | Outside the licensed territory |
| Training a non-ERNIE model on its outputs | No | Distillation and synthetic data make a Derivative Model, and the license bars improving other models |
| Military purposes | No | Acceptable Use Policy, item 19 |
The work, phase by phase
The project runs in four phases over about six months, each ending at a gate with a measurable pass condition.
- Verify the base (weeks 1–2). Run an independent benchmark check against a comparable base model, measure inference throughput for planning RL, and set numeric targets for the later gates.
- Gate: the base scores hold under a neutral harness.
- Build the trainer (weeks 3–8). Port the six custom mechanisms into a training framework, prove the trainer produces the same outputs as Baidu’s inference code, and test the optimizer and warm-up choices.
- Gate, and the real go/no-go: outputs match, and training runs at least 10 million tokens per GPU-hour, the rate this budget assumes. At 8 million, the mid-training budget grows by a quarter.
- Mid-training (months 3–4). Code and STEM at 128K context, then long context stepped up to 256K and 1M, with 15–20% general data mixed back in against forgetting.
- Gate: no loss of general knowledge, and long-context retrieval targets met.
- Post-training (months 4–6). Supervised fine-tuning, then reinforcement learning on verifiable math, STEM and code tasks and on multi-step agentic coding, then preference and safety tuning.
- Gate: the coding and long-context targets set in phase one, plus safety and license review.
Until a trainer reproduces Baidu’s model and runs fast enough, every later estimate is a guess. That is why the second gate is the one that decides funding.
What it costs
| Phase (H100-hours) | Lean | Standard | Lab-scale |
|---|---|---|---|
| Trainer build, parity tests, ablations | 5,000 | 10,000 | 25,000 |
| Mid-training | 18,000 (150B tokens) | 80,000 (600B tokens) | 200,000 (1.5T tokens) |
| Post-training | 24,000 | 90,000 | 280,000 |
| Contingency for failed runs (20% of the rows above post-training) | 5,000 | 18,000 | 45,000 |
| Total | 52,000 | 198,000 | 550,000 |
| At US$2–3.5 per GPU-hour | $100k–180k | $400k–690k | $1.1M–1.9M |
- The standard mid-training mix is about 220B tokens of repository-level code, 160B of STEM, 120B of long-context data and 100B of general text. The 480B at ordinary lengths costs about 48,000 hours; the 120B of long context costs about 2.5 times as much per token, about 30,000.
- Post-training is mostly reinforcement learning. In the standard plan, about 30,000 hours go to RL on verifiable tasks and 50,000 to agentic coding, already counting two or three runs each. Generating the model’s attempts costs about as much as training on them.
- These budgets buy a process, not a guaranteed score. The token counts are round planning numbers, not derived from an ablation. How strong the result is depends on the data and is only known at the gates.
- For scale, pre-training this model took about 2.5 to 3.5 million H100-hours, 13 to 18 times the standard plan.
How we estimated it:
- Cost per token = 6 × active parameters = 6 × 14B ≈ 8.4×10¹⁰ operations.
- Throughput: 8 to 15 million tokens per H100-hour, or about 20–35% of the H100’s BF16 peak. Planned at 10 million, about 100 H100-hours per billion tokens.
- The high end scales DeepSeek’s reported 2.788 million H800-hours for 14.8T tokens at 37B active parameters down to 14B active. The low end assumes a newly ported trainer.
- Long context costs 1.5–2 times as much per token at 128K and 2.5–3 times at 512K to 1M.
- These are estimates from published specifications. The trainer build replaces them with a measured number.
Team and hardware
| Team | People | Owns |
|---|---|---|
| Training infrastructure | 3–5 | The trainer port, FP8 training, context parallelism to 1M, parity tests |
| Data | 3–6 | Code with tests, verifiable STEM, long-context data, decontamination, license provenance |
| Evaluation | 2–3 | Gates during mid-training, long-context and code suites, contamination checks |
| Post-training | 3–5 | Fine-tuning data, reward models and verifiers, the RL system and coding sandboxes |
| Safety and compliance | 1–2 | Safety evaluation, red-teaming, license obligations |
Full-parameter training holds about 4 to 5 TB of state: BF16 weights, FP32 master copies and gradients, and optimizer state (one buffer per parameter with Muon, two with AdamW). That needs at least 16 nodes of eight H100s, with 32 nodes the comfortable size. At 256 GPUs the standard plan is about 32 days of pure compute, which becomes two to three months of calendar time with failures and evaluation pauses. A single eight-GPU H200 node can serve the model for evaluation and RL rollouts but cannot train it.
Risks
- No training code. Six custom mechanisms must be reimplemented, and a subtle mismatch silently degrades the model. Nothing should train until the trainer’s outputs match Baidu’s inference code.
- The optimizer. The model was pre-trained with Muon. Moonshot AI found that “a model that is both Muon-pretrained and Muon-finetuned outperforms others,” and named the “optimizer mismatch” between pre-training and fine-tuning as a barrier. No optimizer state was released, so this needs a small test first.
- An unknown starting point. Baidu didn’t publish where its learning-rate schedule ended. Warming up too hot can undo pre-training.
- No small sibling. Every experiment either runs on 286B parameters or on a small model built to match the architecture.
- Long context to 1M across compressed and sparse attention layers is the least proven piece, so it goes in steps with checks at each. RL at that length is a separate question: the budget assumes RL episodes of at most 128K.
- Forgetting. A heavy code and STEM mix erodes general knowledge without replay and regression gates.
The recommended first step
Fund only the first two phases: about 5,000 GPU-hours, four or five engineers and six to eight weeks, under 3% of the standard budget. They answer the questions that decide whether the rest is worth funding: whether the base is as strong as reported, whether a trainer can reproduce it exactly, and how fast it really trains. Before that, settle the target market, because the license decides who can be served.
Sources: Baidu’s model card and configuration and license (as of November 6, 2026); DeepSeek-V3 Technical Report; Muon is Scalable for LLM Training, sections 3.5 and 4. Parameter count from our sum over the 128 BF16 weight-file headers. All compute and staffing figures are estimates.