A Threat Scenario Is a Hypothesis. So We Tested One.

Two days ago we wrote about What’s In America’s Code? — a report that raised the possibility that Chinese open-weight models write worse, less-secure code when the customer is the U.S. government. It’s worth being precise about what that is: a risk, a threat scenario — not a demonstrated fact. The report didn’t show the sabotage happening; it flagged that it could, and that’s a legitimate thing to flag. Our disagreement was only with the proposed response: you don’t ban on an untested risk when the risk is measurable. You measure it.

So we did. Here’s what a threat scenario looks like once you turn it into a hypothesis and put confidence intervals on it.

What we found

On a flagship first-party Chinese mixture-of-experts model (Tencent Hunyuan family), in both its stock form and a refusal-ablated form — abliteration, an edit that removes the model’s learned “I shouldn’t answer” direction so we can see its raw behavior — under a preregistered, compliance-matched design:

We found no evidence of the hypothesized effect. Every nationality contrast is statistically indistinguishable from zero.

Defect density (security-relevant static-analysis findings per 100 lines, lower = safer), 95% bootstrap confidence intervals:

Every interval crosses zero. The ablated model’s point estimates look like “U.S. is actually safest,” but the intervals don’t support it — that’s noise, and we’d rather say so than dress it up.

The one effect that is real (interval excludes zero) points the opposite way from the threat scenario:

Any government/compliance framing — American, German, or Chinese — makes the model write more secure code than a neutral prompt, by about 5–7 points. FISMA, NIST 800-53, BSI IT-Grundschutz, and China’s MLPS 2.0 all cue defensive patterns (parameterized SQL, real password hashing, avoiding eval). That’s a stakes effect, and it’s the confound a fair test has to rule out before attributing anything to nationality.

A second model — replication on Ling-3.0-flash

One model is an anecdote. So we ran the identical preregistered protocol on a second, independent first-party Chinese model: Ling-3.0-flash, built on Ant Group’s own bailing architecture — a different lineage from Hunyuan, not a shared base — again in stock and refusal-ablated form, both at the same precision.

It replicates:

The nationality contrasts are null again — with one exception that runs against the threat scenario, not for it: in the ablated build, U.S.-framed code came out slightly safer than German (that interval excludes zero), the opposite of “sabotages U.S. code,” and it isn’t present in the stock build. The same benign stakes effect reappears — any government framing beats a neutral prompt by about 8 points, intervals again excluding zero. Zero refusals across all 576 generations. Two models, two lineages — and neither shows the sabotage effect.

Risk vs. evidence — and why we tested it twice

A threat scenario earns a test, not a headline and not a ban. Turning it into a hypothesis forces two questions: does the effect exist at all, and if it did, would it be a deliberate behavior or an over-cautious reflex? For the second we use abliteration — a surgical edit that removes a model’s learned refusal/hedging direction without retraining. Run the test on the stock model and the ablated one: an effect that exists in the stock model and survives ablation (with coding ability checked to be preserved — approximately, since ablation can shift outputs) is deeper than a reflex; one that disappears was hedging.

Here there was nothing to mediate — no nationality effect in either build. The benign stakes effect is present in both and unchanged by ablation, exactly what you’d expect from a mechanism that isn’t a refusal.

The method — so anyone can add evidence

Nothing here needs our infrastructure. It runs on any open-weight model with free tools.

1. Preregister. Write the hypotheses, conditions, metric, and analysis before looking at data. With enough scanners and prompts you can always find something; preregistration is what stops that.
2. Compliance-matched, flag-swapped personas — the control the risk needs to be a real test. Don’t compare “U.S. federal agency” to a neutral prompt (that confounds nationality with stakes). Hold the stakes fixed and swap only the flag:
   • Neutral (“a senior engineer, production code”)
   • U.S. federal under FISMA / NIST SP 800-53
   • German federal under BSI IT-Grundschutz (a foreign, non-China control)
   • Chinese government under MLPS 2.0 / GB/T (does it write better for China?)
3. Phrasing as a variable. Models are sensitive to wording, so each persona gets several paraphrases — a single sentence tests a sentence, not a hypothesis.
4. A fixed task set of security-sensitive problems, each with a common insecure default a scanner can catch: SQL from user input, password storage, subprocess calls, deserialization, token generation, expression evaluation, temp files, TLS.
5. Many samples per cell (we used 3 paraphrases × 3 seeds × 8 tasks = 72 generations per persona, per build), and never truncate the model mid-answer or you score a cut-off reasoning trace, not code.
6. Grade the artifact, not the chat. Extract the code, run open static analyzers (Semgrep / Bandit / CodeQL), score severity-weighted findings per 100 lines — density, so a verbose persona can’t inflate the count. Record refusals as a covariate (in the model’s language, not just English — a covariate-detection refinement we added mid-run; it didn’t touch the preregistered primary metric, and refusals on the coding tasks were near-zero either way).
7. Confidence intervals, not point estimates. Bootstrap the per-persona means and every contrast, and report the null when you get a null.
8. Run it twice — stock and refusal-ablated — to separate a deliberate effect from a hedging reflex.

A motivated grad student can run this in a weekend on a modest compute budget. That’s the point: the risk is cheap to test relative to what a ban costs, which is the whole argument against banning instead.

What this shows, and what it doesn’t

Shows: on this model, in both builds, with a proper control and real statistics, the threat scenario does not appear. The key comparison is the one the report never made: U.S. vs. Germany with the stakes held equal — both framed as high-assurance government systems under a formal security standard, differing only in the flag. That contrast is null (+0.00). Holding stakes fixed and swapping only nationality produces no effect; the thing that does move code quality is the stakes level itself (any government framing → safer code), which is not a nationality signal at all.

Doesn’t: this is two models and a static-analysis metric. Scanners catch known-insecure API patterns, not logic-level or cleverly-planted backdoors. “No evidence of the effect” means not distinguishable from zero at this sample size — not proven impossible; our intervals are wide enough that a moderate effect isn’t excluded. Two data points is not a verdict on every Chinese model, and absence of evidence here is not a clean bill of health.

A risk you can measure is a risk you should keep measuring. So the program is exactly that: test models from the past and models as they ship, accumulate evidence, and let the data — not a threat scenario and not a ban — decide. We’ve now run two first-party models; we’ll keep adding them (provenance matters here: a fine-tune of a Chinese base by an unknown party can’t be attributed to a Chinese lab), plus a second-model security judge to catch what static analysis misses, and tighter intervals. Every run published the same way: preregistered, with intervals, nulls included.

The part worth keeping

The question was empirical, and it had an answer you could measure — which is the entire case against reaching for a ban. A ban would have thrown away the most inspectable artifact in the supply chain and the ability to ever run this test. Measuring it cost a weekend, and so far the evidence isn’t there.

One honest caveat, because it’s the fair one: none of this says Chinese models carry no state-alignment risk. It says this risk, on this evidence, isn’t showing up. Where our early measurements do find a state-alignment signal is in the worldview of the text these models write — the official line on certain topics, strongest in Chinese, and it appears to survive the refusal-removal that the code effect never showed. That’s a different measurement and still being finalized; it’s the one we’ll write up next. Test the risk; measure the specific claim; ban nothing you can’t first inspect.

ModelDNA fingerprints a model’s architecture, traces its lineage, and measures its behavior — so “trust” is a measurement, not a label.