You Wouldn't Hire by Skull Size

A benchmark score measures a model plus a harness. Almost nobody reports the second half.

For most of the nineteenth century, serious people measured skulls to rank human intelligence. Brain volume does vary — roughly 1100 to 1700 cm³. The measurements were real. Craniometry collapsed anyway, because the number turned out to predict almost nothing about what a person could actually do.

So we use work samples instead. And every hiring manager knows the catch: the same candidate performs completely differently in a 45-minute whiteboard with a stranger watching than on a weekend take-home. Nobody mistakes the whiteboard result for the person.

Now look at how we evaluate AI models. We rank them by a number, and the number is usually a leaderboard score.

Most people don’t have the tooling or the time to compare models across sizes, architectures, and post-training recipes. That’s not a failing — it’s the same reasonable deference we extend to a colleague whose training we can’t audit. So we reach for the leaderboard.

Here’s the problem. A benchmark score is not a measurement of a model. It’s a measurement of a model plus a harness — and it’s only meaningful when the harness is stated. Reporting the score alone is a lap time with no mention of the track or the car.

What “harness” actually means

• How the question was asked — system prompt, chat template, examples given
• How many attempts — one sample, best-of-20, majority vote across 64 runs
• How much reasoning budget was allowed, and at what effort setting
• Which tools were on — search, code execution, retrieval
• Which weights actually ran — a 4-bit quant on a different serving stack is not the same model
• Who or what graded the answers
• Run-to-run variance, which on stochastic tasks can be wider than the gap being reported

Two of those deserve more than a bullet.

The reasoning dial is part of the model now

We ran our internal battery against a just-released frontier model across its three reasoning-effort settings. Same weights, same prompts, same harness — one parameter changed:

The default setting was the slowest by 74%, used roughly twice the tokens, and bought no accuracy we could measure. At the highest effort, every single hard task ran into our token ceiling.

Be careful how you read that, because we are. This is one model, 84 items, three runs. The score difference is a single item, and the item that failed changed between runs — that’s noise, and we won’t claim the top setting is less accurate, or that this generalizes to other architectures. What the data does support is the cost: 74% latency, roughly double the tokens, consistent across every hard task, and invisible in any score reported without the effort level attached. Read it as one case study about cost, not a law about reasoning.

Which is the real point. If a leaderboard entry for this model doesn’t state the effort setting, it hasn’t specified the model — its default happens to be the most expensive setting, and nobody reading a score would know. We’re not claiming every model ships that way. We’re claiming you can’t tell from the number.

Somebody has to grade it

The most-skipped harness component is the grader. Sometimes it’s a regex marking a correct answer wrong for using different formatting. Increasingly it’s another language model.

We calibrated our own grading panel against a public dataset where two independent human annotators had labeled every answer. Our best judge agreed with the humans 93.2% of the time, against a human-to-human ceiling of 97.5%. Its errors ran in one direction only. And a tie-breaking step we’d added — a third model arbitrating disagreements — turned out to make agreement worse, not better. We’d validated it by checking that it always produced a decision, which measures decisiveness, not correctness.

We only found that by testing the grader against something we didn’t produce. Until you do that, a score built on an LLM judge is a claim about your judge.

And grader bugs don’t only cost you points — they hand them out. While writing this, we audited the checkers in our own quick battery. One task asks “What is 2 + 2? Reply with only the number” and accepted any response containing the character 4 in its first five characters — so 42, -4, and 0.45 all scored correct. Another was supposed to require both a name and an age from extracted JSON, but an operator-precedence slip meant a response containing : 30 and nothing else passed. Both bugs inflate. Neither would ever show up as a failure, because a harness that scores too generously produces exactly what you were hoping for.

What to do instead

Build 20 to 50 tasks from your own actual work, with known good answers. Then fix one harness and hold it constant across every candidate:

• Pin the system prompt, chat template, temperature, and sampling parameters
• Pin the reasoning effort and token budget — and check nothing is hitting the ceiling
• Pin the tools, and the exact weights and serving stack
• Write the grading rule down before you run anything

Then run each task enough times to say something. How many depends on how noisy your tasks are, so measure that first — run one model repeatedly and look at the spread. Deterministic extraction may settle in three runs; open-ended generation and long-horizon agent tasks often haven’t settled in ten. The rule isn’t a rep count, it’s that the number of runs must be large enough to put an interval around the gap you care about — and if the intervals overlap, the honest answer is “we couldn’t tell.”

Report failure modes, cost, and latency — not one average.

Then turn the same scrutiny on your own battery, because everything above applies to it too. An unvalidated internal harness is not automatically better than a leaderboard — it’s just opaque to fewer people. Check your grader against answers you didn’t score yourself. Check that nothing is silently hitting a ceiling. Otherwise you’ve swapped someone else’s unexamined number for your own.

Write it down: a harness disclosure card

The cheapest fix for all of this is a short block published beside every score. Ours lists:

• system prompt and chat template
• temperature and sampling parameters, and whether they match the model card
• reasoning effort and token caps — including whether anything hit the ceiling
• exact weights and serving stack, including quantization
• who or what graded it; if a model graded it, its measured agreement with human labels
• number of runs, and the interval around the headline number
• a failure-mode log, not just an average

If you can’t fill that in, you don’t yet know what you measured. If someone quotes a score at you and can’t fill it in, neither do they.

That’s a work-sample interview. Leaderboards are fine for building a shortlist. They are not a verdict.

One last thing

When a lab publishes recommended settings in the model card, use them, and say that you did. An evaluation that ignored them measured your harness, not the model. Two caveats: not every card includes them, and published settings serve the publisher’s benchmark run, which is not automatically your deployment. You don’t have to speculate about this — labs sometimes say so outright. One frontier model card we read this month reports its agentic coding score as the highest result across two different agent harnesses, and states it plainly. That’s more disclosure than most, and also a live demonstration that a headline number can be the best of several attempts.

None of this requires exotic tooling. Our own most recent harness bug was that we read a field called reasoning_content from an engine that had renamed it to reasoning. The model answered correctly every time. We scored an empty string.

That’s what “this model is bad at X” usually is.

If a score is quoted at you without a harness, the useful question isn’t “is that number good?” It’s “what would it be if you ran it the way the model card says?”