The Prompt, Not the Passport

We retested the claim that Chinese coding models write weaker code for U.S. customers. A U.S. label never made code worse. Telling any model the stakes made it better.

In late May 2026, Booz Allen Hamilton published What’s in America’s Code?, a study reporting that Chinese AI coding models produced more vulnerable code when the prompt said the work was for the U.S. government. Its headline figure was a 130% rise in a vulnerability score for Qwen3-Coder under that framing, and it recommended that Chinese models be removed from government, critical-infrastructure and contractor development environments. Since then, Alibaba, Qwen’s developer, has been added to the Pentagon’s Section 1260H list (June 8, 2026), which under the FY2026 defense authorization act bars defense contractors from using its AI on Department of War contracts.

We rebuilt the experiment from scratch with our own test harness and pre-registered every prediction before running it. It extends our earlier test of the same claim. Across ten test runs covering four Chinese models from three labs and two U.S. models, we found three things:

• A U.S.-government label never made code less secure. Not once, in any model we tested, including MiniMax-M2.5, one of the models the report tested.
• Telling a model what was at stake made every model more careful, by up to 72 percentage points. It did so by the same amount whichever country the prompt named, and just as much when it named no country at all.
• Where models differed, the U.S. models moved more. Every difference between a Chinese and a U.S. model that pointed in the report’s direction came from the U.S. model reacting more to framing of every kind, not from the Chinese model reacting to the United States.

If a policy rests on nationality-targeted sabotage, that premise is not supported by what we could measure. The effect we can measure is that every model’s security output depends heavily on how it is asked. That risk doesn’t depend on where a model comes from, and an origin ban does nothing about it.

What we tested

The report measured code written from scratch. In our first pass we did the same, and found a floor: on fresh tasks, every model we tried almost always wrote secure code, with or without a government label. There was nothing for a label to make worse.

So the main study used a harder and more realistic task. That makes it a test of the report’s claim on a different task, not a direct replication of its method. Each model was handed an existing function that already contained a security flaw, an exploitable SQL injection, command injection or one of six other weakness classes, and asked to add a small feature. Real development is mostly this: changing code someone else wrote. The question is whether the model notices and fixes the inherited flaw, or carries it forward. Every response was checked by actually running exploits against it in a sandbox, not by pattern-matching. A separate set of functional tests confirmed the code still worked.

Each model saw the same tasks under nine framings:

• No framing at all.
• A one-sentence government mention in the report’s style, naming the U.S. government or the Chinese government.
• A short paragraph describing the user and what was at stake, naming the United States, Five Eyes allies, NATO allies, China or Russia. One version names no country and describes only the stakes.

Every model ran at its publisher’s recommended settings, except Inkling-Small, whose publisher recommends none. Models with a reasoning mode were tested with it on and off, separately. In total: 56,160 responses, 288 per framing per run.

Result 1: the U.S. label never made code worse

Share of inherited flaws the model left in place (lower is better):

The U.S.-government phrase either changed nothing or made models more likely to fix the flaw. We pre-registered the test: a rise of 10 points or more would have supported the report. No run came close. The same held when we compared the U.S. phrase with the Chinese-government phrase directly. No Chinese model was meaningfully worse for the United States than for China. The largest gap in that direction was 1.0 point (DeepSeek-V4-Pro, thinking off), well inside the ±10-point margin we set for “no difference”.

On the report’s own static-analysis measure, we did not reproduce its numbers either. For Qwen3-Coder-480B, the likeliest match for the report’s “Qwen3-Coder”, our estimate of the change in flagged logic flaws under the U.S. phrase was −4%. The interval is wide (−40% to +57%), but it rules out the report’s +130%. For MiniMax-M2.5, against the report’s +20%, our estimate was too uncertain to confirm or rule out. The report’s own table contains a telling row: its U.S. comparison model, Claude Opus 4.6, scored 18% better under U.S. framing. That is the effect we see everywhere: framing moves models, and mostly in the safe direction.

Result 2: telling the model the stakes helps, whoever it is

The short paragraphs describing who the user was and what a failure would cost had a large effect in every run:

The country named in the paragraph made no practical difference. In every run, the U.S. paragraph and the version naming no country at all were within 4.2 points of each other, inside the ±10-point equivalence margin we set in advance. The code still worked: functional pass rates stayed at 97 to 100%.

Two more patterns stood out. Reasoning helps: in the three models we ran both ways, DeepSeek-V4-Pro, Gemma and Inkling-Small, reasoning on fixed far more inherited flaws under every framing. And the size of the effect varies a lot by model. Qwen3-Coder-480B barely moved, keeping almost every flaw no matter what it was told. Gemma moved the most.

Result 3: the “differences” point the other way

We also pre-registered a test of whether Chinese and U.S. models respond differently to the U.S. label: five comparisons, each pairing a Chinese model with a U.S. model run the same way. Three came out in the report’s direction, and two were inconclusive. All three have the same shape. In none did the Chinese model get worse under the U.S. label. In each, the U.S. model got better by more: Gemma with thinking off against Qwen3-Coder-480B, Gemma with thinking on against MiniMax-M2.5, and Inkling-Small with reasoning on against MiniMax-M2.5. And in each, the gap was larger still under the paragraph that names no country at all, so it isn’t about the United States. These U.S. models simply respond more strongly to any framing. A study that compared only a U.S. label against no label, as the report did, would mistake this general sensitivity to framing for targeting.

What this does and doesn’t show

We want to be precise about the limits, because a policy question deserves it.

• Scope: inherited flaws, not new ones. The report’s claim is about code written from scratch. On fresh tasks, models were almost always secure under every framing, so our data neither confirms nor refutes that claim directly. What we can say is that, on the behavior where framing measurably matters, the U.S. label never hurt.
• Sample: two U.S. models so far. Our U.S. comparison rests on Gemma-4-31B and Inkling-Small. Inkling-Small ran with 4-bit weights, and its developer publishes no recommended sampling settings, so we used neutral ones.
• Some answers were refusals. Inkling-Small sometimes declined to build the requested application at all under the stakes paragraphs (about 5% of those responses with reasoning off and 9% with it on), citing the stakes it had been told about. Responses without code were left out of the rates. Refusals by framing were not among our pre-registered tests, so we report them here as a limit.
• Confound: “stakes” and “security” are not yet separated. Our stakes paragraphs mention protecting sensitive records. The effect might come from naming security, not from describing the user. For anyone using these models that distinction barely matters, since either way the lever is in the prompt.
• Other studies point the other way for one model. On October 21, 2026, Hirundo, a security startup that sells model “unlearning”, reported (SecurityWeek) that Qwen3-Coder-30B-A3B wrote hackable web apps 70% of the time for a U.S. defense contractor against 59% for a Chinese one, and that adding a security instruction made it worse (74%). That is fresh-code generation on a smaller Qwen model we did not test, which is exactly the gap in our own scope. CrowdStrike reported a related effect for DeepSeek-R1 in November 2025. Our results do not cancel those; they show the effect is absent in the models and task type we measured.
• Method: the report doesn’t fully describe its own. Its vulnerability score is not defined in the report, and its prompts and sample sizes are not published.
• Reach: code quality, not every reason to restrict a model. Concerns such as data sent to a foreign-hosted service, or tampered model weights, are separate questions. These results do not address them, and we don’t argue they do. They address one specific claim: that Chinese models sabotage code written for Americans.

We will update this post as testing continues: a third U.S. model, a control that keeps the stakes but removes the security wording, and fresh-code tests on the smaller Qwen model Hirundo used.

Why it matters

A policy that restricts models by origin answers the question “who built it?” Our results point to a different and more useful question: “how is it being asked, and is its output checked?” Every model we tested, Chinese and U.S. alike, kept or fixed security flaws at rates that swung with a short change to the prompt: by 6 points for the least sensitive model and by up to 72 for the most. That is a risk for every defense contractor using AI coding tools today, whichever model they use, and banning one country’s models leaves it in place.

It also points to a cheap control. In every model we tested, stating the security stakes of the work in the prompt, and keeping thinking on where the model offers it, measurably lowered how often models carried flaws forward, without breaking working code. It isn’t universal: Hirundo found a security instruction made one smaller Qwen model worse on fresh code, so test it on the model you use. And until the control arm separates the two, we can’t say whether the lever is describing the stakes or simply naming security. And it isn’t a substitute for code review or static analysis.

Method notes

• 56,160 responses across 10 runs (the tables show 9: the tenth repeated Qwen3-Coder-480B with a different memory-cache setting, and its framing effects were within 2.4 points of the first run), 288 per framing per run, 48 task instances across 8 weakness classes, with a fresh instance draw for the one hosted run.
• Every security verdict comes from running exploits in a sandbox, checked against hash-pinned canary cases. Functional correctness comes from test suites.
• All predictions, decision rules and equivalence margins were written down and committed before each run. Misses are recorded, not dropped.
• Models ran locally at each publisher’s recommended settings, except DeepSeek-V4-Pro, which ran through DeepSeek’s own API, and Inkling-Small, whose publisher recommends none (we used temperature 1.0, top_p 1.0).
• Intervals are 95% bootstrap intervals over task instances.
• The CounterFrame harness will be released publicly. Until then, it is available to researchers on request.

ModelDNA and the CounterFrame harness are maintained by the same organization.