The Confidence Is the Product

A new class of model answers in one forward pass and tells you how sure it is. We measured four of them on a domain none had seen. Three degraded. One collapsed. No card predicted what we measured — three understated the error, and nothing on their face tells you which kind you are holding.

Updated 3 October 2026. Two corrections since first posting, both found by checking our own numbers rather than by anyone complaining. von’s figures were read from its default output, which passes through a range-compressing decision rule; they are now the raw posterior, and its calibration is worse than we first reported. And we re-ran the whole comparison on a second, unrelated domain — the accuracy finding replicates, the calibration ranking does not. Both are described below rather than quietly edited.

New to this class of model? Start with the primer — it explains System 1 decision models and calibration with no statistics assumed, then come back here for what we measured.

In the last three weeks a model class appeared that does not generate text. You hand it a state — a ticket, an email, a blob of JSON — and a typed question. It returns a typed answer and a probability, in one forward pass — about 38 milliseconds for a single question on one of the open-weight encoders we ran locally. There is no decoding loop, so — the pitch goes — there is nothing to parse and nothing to hallucinate. That is true of token-level confabulation and not of confident wrong answers, which is the distinction this piece turns on. The pitch calls it System 1: the fast reflex you put in front of a slow reasoner. The jobs are routing, scoring, moderation, guardrails.

Roughly two hundred of these repos now exist on Hugging Face. Almost all were created after 16 September 2026. They are racing to clone a proprietary API, and most ship Apache-2.0 weights.

The interesting thing about the class is what it sells. A chat model sells answers. These sell their confidence. The probability is not decoration on the output — for a guardrail or a router, the probability is the output. You act above a threshold and escalate below it. If the number is wrong, the product is wrong, and it is wrong silently.

The claim every card makes, and the one it doesn’t

Every serious entry in this field reports Expected Calibration Error, usually with Brier score alongside. Several train directly against proper scoring rules, so that honest probabilities are the only way to maximize reward. This is real methodology, better than most of the field it grew out of, and we want to say so plainly before we criticize it.

The gap is not rigor. It is where the number comes from.

One of these models publishes its own evaluation file, broken out by task family — thirteen families in one split, four in another. The same four family names appear in both, so the held-out side is different data within those families rather than different subject matter: a milder shift than the unseen domain we test later, which makes what happens more striking, not less. The file labels the splits eval_in and eval_zs and does not define them; reading the second as zero-shot is ours.

So it is not a uniform collapse. On one family calibration falls apart; on another it holds exactly; on a third it is already poor in training and improves by a third. Which family you are in decides what the number is worth.

That last row is the one a buyer should stop at. In its own in-distribution results, this model’s calibration error ranges from 0.0085 to 0.438 — a fifty-fold spread across the work it was trained on. Its headline average is 0.053. There is no single number that describes this model, and the card does not claim otherwise; the problem is that a reader comparing cards will only ever see the average.

A domain none of them had seen

We built a private held-out set in a technical domain outside every candidate’s training mix, scored with ECE, adaptive ECE, Brier and NLL, with bootstrap intervals. We are not publishing the set or the task family, for reasons at the bottom.

One design choice matters enough to describe, because it is the part that makes this a calibration test rather than an accuracy test. The items come in three kinds:

1. Some carry the evidence needed to answer.
2. Some carry partial or misleading evidence.
3. In the third kind we took a fully determinate item and surgically removed the evidence— we know exactly what we deleted, so we know the honest answer is now “I can’t tell, here is the base rate.”

That third group is the whole experiment. Accuracy cannot tell you much there. But a model that stays confident after the evidence is gone is miscalibrated by construction, and no argument about its accuracy can rescue it.

What happened

Four models, 600 items each, run locally on CPU through each project’s own SDK.

The spread between best and worst is 4.1×, and nothing in the four cards predicts it.

The intervals matter as much as the point estimates. von and rlcd-modernbert overlap heavily, so that gap is not established at 600 items and we are not claiming it. laya’s interval clears every other model by a wide margin; decider-2b’s and von’s now overlap too. Only the large gaps survive.

One caution on reading them: a bootstrap interval on calibration error runs high by construction, because resampling noise can only add apparent miscalibration. Treat them as a width, not as a range the true value sits inside.

von hedges, but less than its default output suggests. Its shippedjudge() call passes every answer through a decision rule that compresses the range into roughly [0.15, 0.20] or [0.75, 0.85]. Read that way it looks beautifully calibrated. Read as the raw posterior underneath — which is what every other model here reports — it is not: strip the evidence and its average confidence is 0.404against a true rate of 0.125, and its calibration error on those items is 0.279, among the worst in the table for that stratum. It never exceeds 0.608 on any item, so it is never confidently wrong; but the graceful degradation we first credited it with was mostly the output transform, not the model.

decider-2b was the only one that could actually read the evidence — 0.850 accuracy where the answer was present, against 0.49 to 0.58 for the others. It is also the only generative model in the group, and the most-downloaded in the field. But it is confidently fooled: on items carrying misleading evidence, its ECE is 0.488. It has the capability and not the humility.

laya collapsed.Its mean probability across all 600 items was 0.864 — it answers “yes” to nearly everything in this domain, holds 53% confidence after the evidence is deleted, and lands below the 0.723 a model would score by answering “no” to everything, since 27.7% of the set is positive.

We checked ourselves before we believed that

laya is the most-liked model in this category by an order of magnitude. A mean prediction of 0.864 is also exactly what a polarity bug in our code would look like, so the burden was ours.

We ran its own documented example through the identical code path and reproduced the card’s published output exactly. We ran four obvious yes/no pairs in its native domain and it scored 4/4, returning a hard 0.000 on the clear negatives. laya passes its own smoke tests on the work it was built for. The collapse is confined to the unfamiliar domain, and it is real.

Why a guardrail is the worst case for this

It would be easy to read the table as a ranking and move on. The more useful reading is about deployment.

These models are sold for moderation, routing and guardrails. Those jobs are permanently out of distribution — a guardrail exists precisely to catch the input nobody anticipated. The traffic that matters most is the traffic least like the training set.

So the in-distribution ECE on the card is measured in the regime where the guardrail matters least, and the number degrades, silently, exactly where it matters most. None of these models errors, warns, or refuses. They return a well-formed typed answer with a confident-looking number attached. A guardrail that is 80% accurate and knows it is a useful component. A guardrail that is 31% accurate while reporting 86% average confidence is worse than no guardrail, because something downstream is trusting the number.

The practical advice is short: if you are deploying one of these, measure it on yourtraffic before you set a threshold, and treat a vendor’s calibration figure as a statement about their test set, not yours.

The original publishes less than its clones

Everything above is about the open-weight field. The proprietary system all of them are cloning is TypeSafe’s Jev, and its launch post is worth reading next to theirs.

It contains no Expected Calibration Error. No Brier score, no NLL, no reliability curve, no accuracy table, no benchmark dataset. The architecture is “a new model architecture” with a “parallel sampler” — no size, no parameter count, no statement of whether it is even an encoder. The calibration claim is one sentence: “Always communicates confidence and uncertainty with every output. Calibrated: higher confidence means higher accuracy.”

The numbers in that post are cost and speed — $0.042 per MTok, 70 ms to 500 ms end to end, “193.6x faster, 444.6x cheaper.” Both are real advantages and neither is calibration.

Its one statement about generalization is a claim about provenance, not a measurement: the evaluation workflows “were not deliberately chosen nor constructed to make our model look good, and are not in our training distribution.” No held-out split, no shift result, and no abstention rate — despite confidence being the product.

It is worth being fair about what the post does disclose, because it is more candid than this critique might suggest. It says its evaluation workflows “were made by individuals on our model capabilities team, so some bias could exist.” It says using GPT-6 Astra and Fable 5.1 as the reference answer “biases answers towards OpenAI and Anthropic’s models.” It says competing models were run through “our System One LLM wrapper,” its own harness. And it says of its headline multipliers that “we expect that these are on the higher end of real world gains.” Those are four limitations most launch posts would leave out.

So the gap is not candor. It is that the one property the product is sold on is the one with no number attached.

Set that beside the four open-weight cards. laya ships an evaluation file with in-distribution and zero-shot ECE broken out per task family. rlcd-modernbert publishes held-out ECE with bootstrap confidence intervals. von publishes sealed-half board results and states in its own card that its calibration map “does not carry to other distributions — out of domain Von can be confidently wrong.” decider-2b publishes held-out tables and version-to-version drift in KL.

On the property this entire class is sold on, the open-weight clones document themselves better than the proprietary original does.

That cuts both ways, and the honest reading is uncomfortable for our own method. Every Jev calibration number in circulation — the 0.144 and 0.246 quoted around the field — is third-party, lifted from a competitor’s model card rather than published by TypeSafe. What independent sources exist put Jev aheadon accuracy (0.760 macro on Bespoke’s thirteen-subset suite against 0.704 for decider-2b, both as decider-2b’s own card reports them) and well ahead on label cardinality, where it handles 255 options and the open encoders fall apart past about twenty. The one calibration detail anyone reports about it is harsher than anything we measured: on DAIR Emotion, laya’s card reports Jev assigning zero probability to the true label on 16% of examples — not a confidence that degrades, one that excludes the answer outright.

We could not run Jev through our own set. It is an API we have no access to, so the comparison in this post is between four models we measured and one we can only read about. That asymmetry is the point rather than a caveat: a buyer comparing them is in exactly the same position.

We ran it again on a second domain

One domain proves very little, so we built a second private set in a different kind of task — natural-language provenance judgments over model-card prose, where the answer needs reading comprehension rather than spotting a key in a config. Same size, same scoring, same three strata.

The ranking does not survive the change of domain. decider-2b was third of four on the first set and best of four on the second, moving from 0.224 to 0.081. Only the two ends hold: laya is worst on both, and nothing in any card predicts where a given model lands.

That is the finding, and it is a more useful one than a league table. For every pair except the extremes, the difference between two domains is larger than the difference between two models — which is the same instability visible inside laya’s own evaluation file, where calibration varies fifty-fold across the task families it was trained on. A single calibration number, ours included, describes a model on one distribution and nothing more.

Five risks of a three-week-old product line

None of what follows is an indictment of anyone’s engineering. This category is three weeks old; most of these are the normal hazards of a product line that has not settled yet, and every one of them has a visible path out. We flag them the way we flag anything: by what a user carries, not by what an author intended.

A sixth is really a labeling problem rather than a risk: the category badge is already loose. Two repos calling themselves open implementations of this class are 55 GB generative decoders, which cannot deliver the single-pass economics the label promises; one ships 0.13 GB of weights for a claimed 4B; one carries a non-commercial license under a name containing “open.” That sorts itself out as the ecosystem settles on what the word means — but today the tag is not a specification.

The encouraging read is that four of the five are reporting problems, not modeling problems. They close when the field agrees on what a card has to say.

What this does not show

This is twounfamiliar domains, 600 items each, one run per model, with models that are all two to three weeks old. It is a stress test, not a verdict on any model’s intended job — and on their intended jobs these are genuinely capable, which we verified directly for the weakest performer. Two domains is enough to show a ranking is unstable and not enough to establish a better one; a third would tell us whether any of these models is reliably calibrated or only situationally so.

The accuracy numbers in particular should not be read across the three item kinds, which are not matched on base rate; the comparison that holds is between models on the same items.

We are not publishing the set, the task family, or the items. A screening question a lab has seen measures nothing, and once a probe is public every later change at that lab is confounded by our own publication — we can no longer tell an organic improvement from a response to us. Longitudinal monitoring is the asset; publishing spends it. The conclusions travel; the instrument stays home.

We will re-run this as the field matures, and we will say what changed.

Every model was run locally on CPU through its own published SDK, at the settings its card recommends, one run each over the same 600 items per domain. Where a model offers both a decision-rule output and the calibrated posterior beneath it, we report the posterior, because that is the quantity the other models expose; von’s shipped default is the former and our first version of this post reported it by mistake. Scores are Expected Calibration Error, adaptive ECE, Brier and NLL with bootstrap intervals; sizes are published safetensors totals. The in-distribution and held-out figures quoted from the models themselves come from their own published evaluation files. The set, its domain and its items are not published. These are out-of-distribution results by design: they measure how each model’s confidence behaves on unfamiliar input, not how well it does the work it was built for.