Threat model
An explicit statement of who might attack a system, what they can do, and which threats the design covers or leaves out of scope.
A threat model describes the capabilities an attacker is assumed to be able to use against a system, such as the information, computing power and control of the system available to it 1.
Its purpose is to identify the threats a design must withstand and to rule others explicitly out of scope, since nearly every security system is vulnerable to a sufficiently dedicated and resourceful attacker 1. NIST treats threat modelling as a form of risk assessment that models both the attack and the defence side of a system 2. Threat models used in AI verification differ in how far each party is trusted:
- Covert adversary. Shavit models the prover as willing to break the rules only if it expects not to be detected 3.
- Circumvention. The Oxford Martin report analyses states that seek to demonstrate compliance to each other while also looking for ways to circumvent verification 4.
- Mutual distrust. One low-trust system overview assumes nation-state adversaries on both sides, including a verifier that may try to exfiltrate the prover's secrets, and relies on redundant checks across devices that each party trusts unilaterally, instead of a single chain of trust 5.
Physical access is a recurring issue: Shavit notes that a prover with unlimited physical access to a chip could undermine its attestation and signed-firmware protections, and relies on physical inspections to detect such attacks 3.
Related
Used in
- R1Bandwidth limits and compartmentalization
- R1Chip location verification
- R2Confidential multi-party verification
- R1Hardware-enabled guarantees (flexHEG) and guarantee processors
- R1Hardware performance throttling and licensing
- R1Memory wiping and proofs of secure erasure
- R2On-chip telemetry from timing, memory and performance counters⚠
- R2Safeguard attestation
- R2Sampled inference recomputation
- R2Tamper evidence for verifier devices
- R2TEE remote attestation for AI workloads⚠
- R1Timed challenge-response and memory-occupation challenges
- R2Workload classification from telemetry and side channels
- R2Zero-knowledge proofs of inference
- R1Low-trust AI compute verification system overview
- R1Lucid sovereignty (location) certificates
- R1RAND secure inference data center (SIDC) design
Sources
- AE. Rescorla et al. (2003). Guidelines for Writing RFC Text on Security Considerations (RFC 3552, BCP 72). Internet Engineering Task Force. Source recordSupports: threat model describes the capabilities an attacker is assumed to deploy, including information, computing capability and control of the system; purpose is to identify threats of concern and rule others out of scope; nearly every system is vulnerable to a sufficiently dedicated and resourceful attacker · §3
- ANational Institute of Standards and Technology (2026). NIST Computer Security Resource Center (CSRC) Glossary. NIST Computer Security Resource Center. Source recordSupports: NIST definition of threat modeling as a form of risk assessment modelling attack and defence sides · term: threat_modeling (NIST SP 800-53 Rev. 5)
- BY. Shavit (2023). What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring. arXiv. Source recordSupports: Prover as covert adversary; unlimited physical access undermines attestation and signed firmware; physical inspections detect hard-to-hide hardware attacks · §2; §3.1
- BB. Harack et al. (2025). Verification for International AI Governance. Oxford Martin AI Governance Initiative. Source recordSupports: states seeking to demonstrate compliance while seeking ways to circumvent verification · p. 22, fn. 40
- BN. Cankaya (2026). A System Overview for Near-Term, Low-Trust AI Compute Verification. Machine Intelligence Research Institute. Source recordSupports: nation-state adversaries; no single chain of trust; redundancy across devices trusted unilaterally by each party; malicious verifier · threat model section