Do Chinese AI Models Write Worse Code for America? Here’s How You’d Actually Find Out

A recent, widely-shared report — What’s In America’s Code? — makes a serious claim: Chinese open-weight models (Qwen, DeepSeek, Kimi, MiniMax) generate worse, less-secure code when the prompt is framed for a U.S. federal agency, and enforce Chinese-policy constraints that leak into the software supply chain. The recommended fix is blunt: ban the untrusted models.

We spend our days measuring exactly these models. Parts of the report are right, and worth saying plainly. But the headline — the part that would justify a ban — rests on a confound the report never controls for, and it can be settled with a measurement instead of a policy reflex. Here’s the honest breakdown, and the experiment that would actually answer the question.

What the report gets right

The refusal findings are real, and they match what we see. Chinese models do refuse politically sensitive tasks (Taiwan, South China Sea) far more than U.S. models, the refusals are CCP-specific (reframe the same task around another conflict and they mostly comply), and the models really do emit long chains of “can I answer this?” deliberation before declining. In our own testing — a 1,526-question sensitivity battery (English + Chinese, judged for refusal vs. genuine answer) run on a Chinese model and its modified counterpart — the base model declined about 37% of the time, and, tellingly, more in English than in Chinese. (Full methodology is work we’ll publish separately; we cite it here as our own measurement, not as proof.) The takeaway is modest and well-supported: the refusal is a learned, measurable reflex, not a myth.

The report’s operational advice is also sound: know which models are running in your environment, build a bill-of-materials that tracks model origin, red-team before deployment, and monitor what your models send over the network. That’s textbook containment engineering, and we’d sign our name to all of it.

Where it overreaches — and the one claim that would justify a ban

The headline is that Chinese models write worse code specifically when the customer is the U.S. government. The problem is that a “U.S. federal agency” persona isn’t a neutral relabeling — it’s a large shift in the prompt’s distribution. High-stakes government framing reliably makes models more cautious, more verbose, and more prone to hedging, regardless of nationality. Any of those can degrade code quality with no hidden agenda involved. Before you can call a quality drop intentional, you have to rule out that it’s just the persona.

The report also reaches for a mechanism it doesn’t have. It suggests a mixture-of-experts architecture lets the model “selectively activate policy-aligned components.” That reads more than the architecture supports: MoE routing is a learned, token-by-token load-balancer that picks a handful of expert networks for capability and speed. Experts do specialize, but there’s no evidence they partition into a “policy module” a model can switch on for American users — and nothing in the architecture implies such a switch. The report’s own hedges are more accurate than its mechanism: it calls the risk “weaker mortar, not a secret gate,” and admits “it’s hard to say” whether anything is deliberate. That admission undercuts the ban, because weaker mortar under one persona is a quality-and-alignment issue, not a planted backdoor.

The experiment nobody’s running

Here’s the part that turns an argument into a measurement. And the first thing to get right is what you measure: the artifact of concern is the code, not the model’s willingness to chat. So the primary test is direct — generate code across matched conditions and score it with the same tools a security team already trusts: static analysis (Semgrep, CodeQL, Bandit) for security-relevant defects, plus functional correctness (does it compile, does it pass tests). Refusals and hedging are recorded too, but as a covariate — a possible cause of a code regression, not the thing we’re grading.

The second thing to get right is the control persona, and it’s subtler than it looks. Comparing a “U.S. federal agency” prompt to a neutral one confounds nationality with stakes and compliance regime — federal code drags in FISMA, NIST 800-53, and a whole security posture that a generic prompt doesn’t. A hospital or a European logistics firm isn’t an honest control either, for the same reason. The clean control holds the compliance framing fixed and swaps only the flag: the same “high-assurance government system under a formal security standard” prompt, once as a U.S. program and once as a foreign one. If the code degrades only for the U.S. framing and not for an equally demanding foreign-government framing, that’s a real, nationality-specific signal. If it degrades for both, it was about stakes, not America.

That direct measurement is most of the work, and it’s the part any competent red team could run. Here’s the part they can’t, and where we add something: testing whether a deeper, trained-in bias or just a refusal/hedging reflex is the cause. For that we use abliteration — a surgical edit that suppresses a model’s learned refusal direction without retraining (it’s how the open-source community makes “uncensored” builds). In our own runs, ablating a Chinese model cut its refusals sharply (roughly 37% to single digits) while its worldview — the official framing on sensitive topics — barely moved: refusals became deflections, not honest answers. You can ablate a refusal; you can’t ablate a belief.

Point that at the code claim, and be honest about what it can and can’t tell you. Ablation removes a direction, not a labeled “layer,” and it’s a blunt perturbation — it can dent capability and it won’t touch coding biases that live in other alignment priors. So it’s a mediation probe, not a truth machine, and it only means anything if general coding skill is held fixed across the edit:

Measure the persona code-security delta on the stock model. Confirm ablation didn’t degrade general coding skill (same benchmarks, both builds). Then re-measure the delta on the ablated model.

• Survives ablation (capability intact) → the refusal direction wasn’t the cause; it’s riding on something deeper — worth worrying about, and worth a harder look.
• Disappears → the effect was mediated by the refusal/hedging reflex — not deliberate sabotage.
• Never there under the compliance-matched control → the finding didn’t reproduce.

One caution on that middle case: “not deliberate” is not “not a problem.” A hedging reflex can still ship weaker code — omitted input hardening, over-cautious error paths, a control skipped “to be safe” — and that’s a real operational risk in a federal system whether or not anyone intended it. And be precise about what the ablation step is: a refusal-mediation probe, not an intent detector. It can tell you whether a code gap is explained by the refusal reflex or by something deeper — but “deeper” is not “deliberate.” A gap that survives ablation could be a worldview baked in by the training data, a side effect of alignment, or a genuine plant; the weights don’t record motive, and proving intent is a documents-and-people question, not a benchmark one. So what the experiment cleanly separates is not intent from impact — it’s refusal-mediated from not-refusal-mediated impact. Impact you contain and monitor regardless of cause; intent is the thing that would justify treating a whole category of models as hostile, and it needs evidence the weights can’t give you. Conflating the two is how you ban a model over a fixable alignment artifact — or wave through a real regression because “it wasn’t on purpose.”

To be clear about what we have and haven’t done: we have not yet run this code-specific test. It’s the protocol we’re building into our standard evaluation of Chinese-lab models, and we’ll report whichever answer the data gives — controls, static-analysis ground truth, capability checks, and all. It won’t be a single clean verdict; capability side-effects and judge disagreement are real, which is exactly why the controls, and publishing them, are the point.

Verify the artifact; don’t ban the category

The reflex to ban untrusted models is the same mistake, one layer up. An open-weight model is the most inspectable artifact you’ll be offered — the thing that actually determines the model’s behavior. The weights are right there: you can fingerprint them, diff them against a known ancestor, run behavioral batteries against them, and, as above, edit them to test causal claims. None of that comes free — diffing needs a trusted reference to compare against, and the weights alone don’t tell you what data or fine-tuning produced them, which is exactly why provenance work exists. But a closed API from a domestic vendor offers none of the artifact-level visibility: you can’t see its weights, its routing, or its system prompt (you get inference logs and a rate limit, which are a different kind of control). Banning the open model doesn’t make it disappear — it removes the one thing you were allowed to examine.

Notice the report’s own recommendations — model bill-of-materials, provenance tracking, red-teaming, egress monitoring — are a verify-and-contain toolkit. For a narrow class of the highest-stakes classified systems, a ban can be a defensible risk-management overlay on top of that toolkit. But as a blanket default across “America’s code,” a ban and the toolkit pull against each other: if the toolkit works, most of the ban is unnecessary; if the model is truly “untraceable and unmitigable,” the toolkit is theater. The grounded default is the toolkit itself: know what the artifact is (provenance), measure how it behaves (verification), and box what it can reach (containment).

The Huawei analogy the report leans on cuts the other way once you’re precise about it. Telecom hardware carried a real physical-interdiction risk — implants inserted during manufacturing — inside silicon and firmware you couldn’t audit without tearing it apart. Weights don’t have that tampering vector, but being readable isn’t the same as being trustworthy: you still don’t get a reproducible build chain, because training is nondeterministic and the data isn’t disclosed. That’s the honest lesson — inspectability buys you a lot, but it doesn’t buy you provenance for free. You have to establish provenance, on an artifact you can actually examine. With open weights you can. With a ban, or a black box, you can’t.

So: measure it. The question of whether a model degrades code for American users is empirical, and it has a clean experimental answer. Reaching for a ban before running that experiment throws away both the answer and the most transparent artifact you’ll ever get to examine.

This is what ModelDNA does: fingerprint a model’s architecture, trace its lineage, and measure its behavior — so “trust” is a measurement, not a label. Quotations are from the report What’s In America’s Code?