Weight exfiltration
Weight exfiltration is the unauthorized copying of a model's trained parameters, its weights, out of the environment meant to contain them 1 2.
In a study on securing model weights, RAND researchers identified 38 meaningfully distinct attack vectors for stealing frontier model weights, and defined five security levels for defending against actors ranging from opportunistic criminals to highly resourced nation-state operations 1. Exfiltration can be covert. An attacker who controls an inference server could hide weights inside ordinary model responses using steganography 2. In a 2025 paper on detecting weight exfiltration, Rinberg and colleagues bound the unexplained information in outputs by verifying them against a reference. On the MoE-Qwen-30B model, under benign prompts, their detector reduced exfiltratable information to under 0.5% of the outputs' theoretical capacity at a false-positive rate below 0.01%, slowing an attacker more than 200-fold 2 3. Physical side channels offer another route past network monitoring, which side-channel suppression would limit 4. In verification the concern runs both ways. The verifier's equipment could leak the prover's secrets, so Cankaya's low-trust design commits model checkpoints to an independent governing body rather than revealing them, and sends only hashes to the verifier outside the facility 5.
Related
Used in
- Research demoBounding unexplained information in outputs
- Operational useHardware-attested weight binding⚠
- ProposedNetwork taps and certifiers
- Operational useSampled inference recomputation
- ProposedSide-channel suppression for isolated facilities
- Research demoAttestable Audits
- ProposedRAND secure inference data center (SIDC) design
- Model weights have not left the facility
Sources
- Tier B. S. Nevo et al. (2024). Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models. RAND Corporation. Source recordSupports: model weights as a target for theft; 38 attack vectors; 5 security levels; adversaries from opportunistic criminals to nation-state operations · summary
- Tier B. R. Rinberg et al. (2025). Verifying LLM Inference to Detect Model Weight Exfiltration. arXiv. Source recordSupports: exfiltration by hiding weights in model responses (steganography); security game; verification against a reference; on MoE-Qwen-30B, exfiltratable information under 0.5% at a false-positive rate under 0.01% · abstract
- Tier C. R. Rinberg et al. (2025). Defending Against Model Weight Exfiltration Through Inference Verification. LessWrong. Source recordSupports: authors' statement that the <0.5% bound is a share of theoretical capacity, i.e. of the total information sent through the user channel; >200x longer exfiltration timelines · One Minute Summary; Figure 5 caption
- Tier C. N. Cankaya (2026). Suppressing Side Channels in an Untrusted Data Center via Retrofitted Defenses. MIRI Technical Governance Team. Source recordSupports: physical side channels can bypass network monitoring · side channels of concern
- Tier B. N. Cankaya (2026). A System Overview for Near-Term, Low-Trust AI Compute Verification. Machine Intelligence Research Institute. Source recordSupports: malicious verifier may aim to exfiltrate prover secrets; checkpoints committed, not revealed, to an independent governing body; only hashes sent to a verifier outside the facility · threat model; weights