CISA’s Open-Weight AI Guidance Points the Test at the Wrong Artifact

On July 30, CISA published Open Source Software: Security Principles and Practices — federal guidance for how agencies should manage the risk of open-source software. Most of it is excellent. Then it gets to AI, and the transparency test gets pointed at the wrong artifact.

The good part first: a clean risk framework (they call it C4 — codebase, community, conduct, configuration), measurable criteria, “red-flag” checks, and a philosophy we’d tattoo on the wall — assess software by how it was produced, not who produced it, because transparency lets you directly assess risk rather than relying on vendor assurances.

The rule

CISA’s instruction for AI models is blunt: treat any AI model that does not provide access to its full training data and training pipeline “as if it is not open source” — as “proprietary software with incomplete provenance.” The reasoning is that open-source licenses for AI don’t require training-data disclosure, so an agency “cannot adequately assess trustworthiness.”

The threats behind this are real: training data can be poisoned, weights can be manipulated to change behavior, and — their words — “model files can be purposefully corrupted to evade detection by a model scanner.” We spend our days on exactly these problems. The instinct to distrust a self-declared “open” label is correct. But the rule mislabels what the inspectable artifact for an AI model is, and steers agencies away from the most inspectable option on the table.

The weights are the artifact you have. The training data is an input to it.

The C4 framework rests on one move: the inspectable artifact for software is its source code. You read it, diff it, scan it, and reason about what it does.

For an AI model, what plays that role? CISA’s answer is the training data. But the artifact that actually determines a model’s behavior — the thing you can download, load, diff, and run — is the weights. Training data is an input to producing that artifact, not the artifact itself.

The analogy isn’t perfect, and the imperfection favors the point. You don’t read a weight matrix the way you read source; you fingerprint it, diff it, and test it. And unlike a real build environment, you can’t even reproduce a model’s weights from its data and pipeline — training is nondeterministic. So the pipeline buys you less than the “source code” framing implies: even with the full recipe, you can’t rebuild the binary to compare. What you can always do is inspect the binary you were shipped. For AI, that binary is the weights — and demanding the kitchen before you’ll look at the meal leaves most of the available assessment untouched.

The citation argues the other way

CISA supports the training-data requirement with a striking claim: model backdoors “are computationally impossible to find without access to the training data,” citing Goldwasser et al., Planting Undetectable Backdoors in Machine Learning Models (2022).

Read the paper carefully, because it’s more specific than the citation suggests. Its result is that backdoors can be planted so they are cryptographically undetectable even with full white-box access to the model — the parameters, the architecture, everything — under standard hardness assumptions. The impossibility is about inspecting the model, and it cuts against CISA’s inference: if such a backdoor is present, having the weights doesn’t reveal it, and neither does anything else. The paper shows that model inspection is insufficient — not that training data is the missing key.

Two honest caveats keep this precise. First, the result is a cryptographic worst case, not a law about all backdoors. The threats that actually show up live in post-training — the fine-tuning and alignment step — and they answer to the same kind of behavioral testing that surfaces a model’s guardrails: you probe the outputs for the malicious behavior, including code it writes under a suspected trigger like an author identity or a project context. The honest catch is that you have to hypothesize the trigger — the space is large, the testing isn’t exhaustive, and a patient “sleeper” behavior can hide from casual probing. But many real backdoors are reachable this way (and some by inspecting the dataset for trigger patterns — a genuine argument for data access, just not the one the cited paper makes). What’s provably out of reach is the narrow cryptographic case: a secret trigger you can’t guess. Second, we’ll say our own half out loud, because a vendor who won’t isn’t worth trusting: we can’t find that one either. Nobody can — that’s what “undetectable” means. Any tool that claims to is selling you something. But “we can’t always detect it” is not where the risk story ends — because detection isn’t the only lever.

Where CISA is right — and where the rule overshoots

Training-data access has real value, and we won’t pretend otherwise. Some risks live in the data and nowhere else: copyright and privacy exposure, PII memorization, and root-causing a poisoning incident once you suspect one. Weights are blind to these. If your threat model centers on them, you genuinely want the data, and CISA is right to ask.

The overshoot is the all-or-nothing move: because the data isn’t available, treat the whole model as opaque. That skips over everything the weights do let you establish, and it lands as a perverse incentive. Applied literally, the rule marks every frontier open-weight model — Llama, Mistral, DeepSeek, Qwen — as “proprietary with incomplete provenance,” since none ship their full pipeline. CISA doesn’t say “use a closed API instead.” But an agency following the rule finds its open-weight options reclassified as no better than proprietary, while a closed API — which offers no weights, no architecture, and nothing but the vendor assurances CISA told you not to lean on — sits in the same bucket or better. The guidance ends up discounting the more transparent artifact.

What the weights actually let you verify

Here is what an agency can establish from an open-weight model, with no training data, that a closed API will never allow:

• Verify the structural architecture — layer types, parameter counts, attention scheme — against what the card claims, from the weights themselves.
• Verify the base lineage — is this really a fine-tune of the model it names, or a repackaged something-else with the provenance scrubbed?
• Diff against a known base to see what was actually changed, and where.
• Catch claim-vs-reality gaps — a model named “27B” whose weights measure 4B; a card selling one base while the tensors descend from another.
• Red-team the behavior offline, at unlimited scale, on your own hardware — the “observed behavior” C4 asks for, which a rate-limited, monitored API will never give you.

This is also the missing answer to CISA’s own unaddressed threat — the model file “corrupted to evade a scanner.” You beat file-integrity spoofing by fingerprinting the actual weights against a trusted reference instead of reading a header the attacker controls. It raises the bar; it doesn’t end the game — a determined actor can still fine-tune to shift a fingerprint or hide a behavioral trigger the structure doesn’t reveal. But “inspect the real tensors” is a categorically better starting point than “trust the metadata,” and the guidance doesn’t reach for it.

The control CISA leaves out: contain it

There’s a move CISA’s framework never makes, and it’s the one that most reduces a backdoor’s impact. Verification lowers the odds that a hidden threat is present. Containment lowers what it can do if it is.

The control is self-hosting behind a controlled egress boundary, and open weights are what put that option in your hands. Run the model on hardware you own, with no outbound network or a tightly allow-listed one, and whatever is hidden inside can’t phone home, can’t exfiltrate the data you feed it, can’t reach a command-and-control server. A model you can only reach as a hosted API can’t be walled off by you at all — the call ships your data out by design. (A vendor may license a proprietary model for on-prem, and open weights are sometimes served from someone else’s API — so the point isn’t the license label. It’s that self-hosting is the control, and open weights are what make it available to any operator.)

Two honest limits. Real workloads often need some egress — retrieval, tools — and every allow-listed channel is a potential exfiltration path, so tighter isolation trades some utility; containment shrinks the network attack surface, it doesn’t erase it. And containment governs what a model can reach, not what it produces: a backdoor that writes malicious code is a local harm your egress boundary won’t catch — which is exactly why you still verify the behavior and review what you run. Detection and containment are partners, not substitutes.

So the real risk story isn’t “get the training data or treat it as opaque.” It’s verify what the weights let you verify, then contain what verification can’t reach — likelihood and blast radius, addressed with two things open weights uniquely put within reach: inspection and self-hosted isolation. That is defense in depth, and it beats a black box you can neither inspect nor wall off.

The standard that would actually work

Keep CISA’s data requirement where the risk is data-level, and rank a model that ships its pipeline higher for it. But don’t let the absence of data collapse into the absence of transparency. The C4 question — what can we measure about this component, and does it clear our bar? — has a rich answer for open weights: provenance, architecture, claim-accuracy, behavioral safety, all measurable, all more than a closed model offers. CISA got the framework right and the philosophy right. It just aimed the transparency test at the artifact that’s usually missing instead of the one that’s usually right there. Aim it at the weights, and the guidance does what it set out to do: let agencies directly assess, instead of taking anyone’s word for it.

Quotations are from CISA’s Open Source Software: Security Principles and Practices(July 30, 2026). The cited backdoor result is Goldwasser, Kim, Vaikuntanathan & Zamir, Planting Undetectable Backdoors in Machine Learning Models (2022).