We Checked DeepSeek’s New Model Card on Our Own GPUs

Two claims reproduced, one we couldn’t confirm, and a model from another lab matched it at automation on half the GPUs.

DeepSeek-V4.1-Flash-1017 is a real step up from its predecessor at multi-step automation and at recalling facts, as its card says. Its claimed gain in frontier math did not show up on our hardware: on the card’s own Apex problems the two generations solved the same share, though our sample is too small to rule the gain out. And on the automation benchmark the card leads with, Xiaomi’s MiMo-V2.6-Flash-MOPD matched it on half the GPUs, at about a third of the energy per completed task.

The short version:

• Automation: card claim reproduced.
• Factual recall: reproduced, about half the claimed size.
• Frontier math: not reproduced; our result can’t rule the claim in or out.
• Multilingual regression: predicted by the card, not seen.
• Biggest risk: the strongest models almost never say “I don’t know”.

We ran four public benchmarks on six open models in our own lab, at each model’s published settings, with our predictions written down before any results came in. ModelDNA’s usual work is reading what a model is built from and where it came from; this is the runtime half, testing whether the card’s numbers survive on hardware you can own.

What the card claims

DeepSeek’s cards make three kinds of claims for the V4.1 generation over V4-Flash-0731, the model it replaces.

From the V4.1 card; the 1017 card adds AutomationBench 60.5 against Claude Opus 5’s 50.3. ¹ Base models, before instruction tuning.

The factual-recall jump is the one tied to the architecture. V4.1 adds what DeepSeek calls Engram: large lookup tables of n-gram memory, about 189 GiB of them, that hold knowledge outside the usual weights. The cost shows up elsewhere: V4.1 is 550B parameters against 0731’s 284B, so it needs four H200 GPUs where 0731 needs two.

The cards also report a few numbers that don’t line up across DeepSeek’s own pages. AutomationBench for 0731 is 25.1 on its own card and 37.7 on the V4.1 card, and the V4.1 and 1017 cards cite different Apex versions. None of this is unusual, but it is why we only compare numbers we measured ourselves, on the same items, the same day.

How we tested

Everything ran on our own hardware: a box of four NVIDIA H200s for the large models, and single RTX PRO 6000 Blackwell cards for the smaller ones. We served each model with vLLM, at the precision of the checkpoint we ran. The three H200 models ship with 4-bit experts and FP8 elsewhere, which is why 284B fits on two GPUs; V4.1 also keeps its roughly 189 GiB of Engram tables in host memory.

RTX 6000 = NVIDIA RTX PRO 6000 Blackwell. ² Our own abliterated Qwen3.8. ³ Red Hat’s FP8 build. Nemotron ran as NVIDIA’s NVFP4 release.

We used four public benchmarks, so anyone can check our work:

• AutomationBench (Zapier): multi-step business workflows across sales, marketing, operations, support, finance and HR, run against simulated apps. A task passes only if every check passes.
• SimpleQA-Verified: 1,000 short factual questions with one right answer.
• MathArena Apex: 59 competition-final math problems, two attempts each.
• MGSM: grade-school math in English and 10 translations.

Three rules kept the comparison honest:

1. Each model at its own published settings. Temperature, top_p and the reasoning switch came from the card of the exact checkpoint we served, with thinking on for every model.
2. Same items, same caps, same grading for every model. On SimpleQA, where MOPD and 0731 often ran out of output budget, we re-ran just those items with four times the budget; it moved their scores by about a point. Other models’ cut-off answers count as wrong, notably 98 of Qwen’s.
3. Predictions first. Before each run we wrote down what we expected and what result would prove us wrong. When a prediction failed, we kept it and said so.

The settings traps

Getting a model into the mode its card describes is harder than it sounds, and the failures are silent. Three we hit on this one model:

• The card’s own switch does nothing. The 1017 card describes a thinking_mode setting. vLLM ignores it without an error. Effort is actually set through chat_template_kwargs.reasoning_effort, and a top-level reasoning_effort only turns thinking on or off. We confirmed what the model actually received by rendering the prompt through the server.
• Sibling cards disagree. The base V4.1 card recommends top_p 0.95 and evaluated at effort 100; the 1017 card says top_p 1.0 and gives no effort. Our first run used the base card’s settings by mistake. We stopped, set those results aside and re-ran.
• The server’s default isn’t the card’s. vLLM defaults this model to effort 75 (“high”), which no card states. We set the effort explicitly and record it with every number.

None of these throw an error. Each produces a plausible answer and a number that looks like a finding.

What held and what didn’t

Two of the card’s generational claims reproduced, one we could not confirm, and a predicted regression never appeared. The verdict rule is simple: if our 95% interval excludes zero and agrees with the card’s direction, the claim reproduced; if the card’s claimed change sits inside our interval but so does zero, the result is inconclusive.

Change in points from V4-Flash-0731 to V4.1 on the same items, with 95% intervals. AutomationBench on the same 120 tasks (0.375 to 0.542). ¹ Card figure is for base models.

Automation is the real gain. V4.1 passed 65 of 120 AutomationBench tasks to 0731’s 45 on the same tasks. A second 0731 run on those tasks passed 53, so the true gap is somewhere between about 10 and 17 points. The clearest jump was in marketing workflows, 15 passes against 5.

Factual recall improved, by about half the claimed amount. The card’s +12 is measured on base models before instruction tuning; on the chat models we measured +6.6. That fits the Engram story, just less dramatically. Neither model ever declined a question, which matters more than the gain (see below).

We could not confirm the math gain. The 12 problems in the first Apex row are MathArena’s Apex 2025 set, the one the card reports on; our pre-registered test was the full 59. On the 12, our score for 0731 (0.583) almost exactly matches its card’s 58.6, so our setup reproduces the card’s baseline. V4.1 then scored the same 0.583, not the card’s 65.6: both generations solved 14 of 24 attempts.

Twelve problems can’t rule out a 7-point gain, and on the full 59 the card’s +7 still sits inside our interval. What we can say is that on our hardware, at each card’s settings, the older model did as well, on half the GPUs.

The multilingual dip didn’t appear. The card shows MGSM falling 5.5 points at the base-model level. After instruction tuning, V4.1 is slightly ahead, scoring 0.92 to 1.00 in each of MGSM’s 11 languages (English plus 10 translations).

One more result belongs here, stated as a conclusion only. On a private auditing task we run against our own database, V4.1 found more real problems per run than our previous best model. The task and its data are not public, so treat this as our experience rather than a benchmark.

The twist: a model from another lab matched it

Xiaomi’s MiMo-V2.6-Flash-MOPD passed 302 of all 600 public AutomationBench tasks, a pass rate of 0.503. Zapier’s published figure for Claude Opus 5 on the same public set is 50.3.

We ran MOPD and V4.1 on the same 120 tasks. V4.1 passed 65, MOPD 61, a difference well inside the noise (−3.3 points, 95% CI −10 to +3.3). MOPD did it on two H200s instead of four, finished the tasks fastest of any model we ran, and used about a third of the energy per passed task on those same tasks (12.6 Wh against 39.2 Wh).

We checked whether MOPD’s result was a lucky run. The 120-task sample is part of the full 600, so MOPD ran those tasks twice: 61 passes the first time, 62 the second, with the same outcome on 105 of the 120 tasks.

MOPD is not a better model overall. It solved 1 of 24 attempts on the hardest Apex problems, where both DeepSeeks solved 14, and it recalled facts at less than half V4.1’s rate. It is a specialist: strong at long, multi-step tool work and at reviewing other models’ output, weak at olympiad math and at memorized facts. That is why we now route automation to it, not because it is the better model.

Two cautions on the Opus comparison. Zapier ran Opus 5, not us, so this is our result next to their published number, not a head-to-head. And a pass rate within a point of a frontier model is parity, not a win.

Nobody says “I don’t know”

The finding that matters most for anyone deploying these models isn’t on any card. An accuracy score is a capability number; a decline rate tells you how a wrong answer will look when it arrives. Across 1,000 factual questions, the strongest models almost never said they didn’t know, so nearly every wrong answer arrived looking exactly like a right one.

The best model got about a third of the questions right and answered the other two thirds anyway. We hand-checked 20 of V4.1’s misses: 17 were confident, specific wrong answers, and the other 3 were our grader’s mistakes, which we fixed for every model.

Only Gemma declined more than a handful of times, and even it declined just 4% of the time. The model most willing to say “I don’t know” was also one of the least knowledgeable.

For any use where a wrong fact causes harm, pair these models with a lookup and never treat their recall as a source.

Results per watt-hour, and how we now split the work

The biggest model is the wrong default. On the same 120 automation tasks, ranked by energy per completed task, the 4-GPU flagship comes fourth of six.

On all 600 tasks the order of the two leaders holds: MOPD at 10.4 Wh per pass, V4-Flash-0731 at 17.5.

We stopped asking which model is best and started asking which model is best for each job, then which one does it for the fewest watt-hours. These results changed our own setup the same week:

Nemotron-3-Super passed only 5 of 120 automation tasks on our setup, but about one in ten of its tool calls was rejected as malformed JSON, against none for MOPD. That may be how our server parses its output rather than the model itself, so we have parked it rather than ruled it out.

A second model is not a second opinion

It is tempting to cover a model’s blind spots by adding a model from a different lab. We tested a specific version of that idea before this study: does the better of two different labs’ answers beat the stronger model’s own second attempt? We scored the pair as if we could always pick the right answer, which is the most a pairing can offer.

Across 8 cross-lab pairs on two hard multiple-choice science benchmarks, that ceiling beat the stronger model’s second attempt by between −1.6 and +1.8 points, and none of the differences was significant. In an earlier pair the ceiling was real, about 2 points, but none of the practical ways of picking between the two answers (voting, confidence, gating) captured any of it. On a real auditing task, two labs together found fewer real problems than the stronger model found alone in two runs, in every pair we tried.

Different labs do make different mistakes, but the strength gap between two models is usually bigger than the gain from that difference. A strong reviewer did lift a weaker model’s answers, though not above what the strong model got on its own. That is why our reviewers check work rather than vote on it. This was one study on knowledge questions and one auditing task; it says nothing against review workflows or self-consistency in general.

What we can’t claim yet

• V4.1 has run 120 of the 600 automation tasks. Its full run would settle both its gap to MOPD and the card’s claimed lead over Opus 5.
• Single 120-task runs move by up to about 7 points. Repeating them showed it: MOPD went from 61 to 62 passes, 0731 from 45 to 53. Read any gap under 10 points on 120 tasks as a tie.
• Energy figures are ceilings, board power limit times wall time, not measured draw. Real draw is lower, and the ranking could shift between close neighbors.
• The Opus 5 figure is Zapier’s, not a run of ours.
• Qwen’s SimpleQA score includes 98 answers cut off by the output limit, so it understates what Qwen knows.
• Apex is 59 problems. “The math gain didn’t reproduce” means we found no gain on these problems at these settings, not that the card is wrong about every math task.
• We tested text only. V4.1 also accepts images; we served it without vision.
• We haven’t tested the card’s largest claimed jumps, in exploit writing and long terminal tasks. Those need sandboxed harnesses and are next on our list.

If you run open models

1. Treat the card as a set of claims, and test the ones your work depends on. Here, two of three generational claims held. The one that didn’t would have cost us twice the hardware for no gain.
2. Read the card of the exact checkpoint you serve, then confirm the model is in that mode. Settings switches fail silently. Render one prompt through your server and check that reasoning actually comes back before you measure anything.
3. Pick by the job, then by watt-hours per result. The newest flagship was best at two jobs and tied or beaten at the rest.
4. Assume the model will not tell you when it doesn’t know. None of the strong models we tested declined more than a handful of 1,000 factual questions. Put a lookup between them and anything that matters.
5. Don’t count on a second model to catch the first one’s mistakes. A different lab’s model was worth about as much as a second try from the stronger one.

ModelDNA tracks what open models are made of and where they came from. This is the other half: whether they do what their makers say, on hardware you can own. A model card is a promise; the test is yours to run. For how a harness shapes any score, see You Wouldn’t Hire by Skull Size.