The claim is that outputs delivered to users or auditors come from the specific model, weights and configuration the provider declared.
Evaluations, audits and agreements often apply to one specific model. If a provider could evaluate one model and serve another, such as a cheaper, quantized or modified version, those checks would say little about what users receive. It is a positive claim that can be tested directly, but three problems make it hard.
Numerical nondeterminism means honest recomputation does not match exactly. The verifier usually cannot see the weights, which are commercially or strategically sensitive. And the evidence must come from the actual serving system rather than a separate test instance.
Approaches include statistical or exact recomputation of sampled outputs, hardware attestation of the loaded weights, and zero-knowledge proofs. They trade off cost, trust in hardware vendors and confidentiality.
The served model can be checked directly on its outputs, and several mechanisms for it have reached research demonstration (R2) or operational use (R3). These development statuses are separate from security evidence and do not show that a mechanism is secure. Known attacks and limits each hold under their own conditions. Forging Intel TDX attestations through the memory bus needed physical access to the hardware 13, but another attack forged AMD SEV-SNP attestations from a malicious hypervisor, in software alone, on platforms without AMD's fix 15. One statistical test of API outputs reached a median power of 77.4% against a range of distortions, using an average of 10 samples per prompt 16. Power here is the probability that the test detects a specified change to the model under the tested conditions 16.
For example, users of a hosted model may want to know that the model answering them is the one an auditor evaluated.
- Hardware-attested weight binding (R3) extends TEE remote attestation (R3) from the software an enclave booted to the weights it loads. Confidential multi-party verification (R2) runs the audit in an enclave and binds its result to the model's hash 10.
- Attestable Audits specifies how to join the two steps. At inference, the enclave checks the served model's hash against the earlier attestations and returns each response with an attestation that links model, prompt, output and audit result 10. Its R2 rests on the audit step alone, on CPU-only enclaves with a 4-bit, 8-billion-parameter model, and the inference step is unevaluated 10. Tinfoil's Modelwrap (R3) covers serving alone, in what Tinfoil reports is a production service 12.
- The answering model need not be the audited one. In Attestable Audits an enclave can quantize the full model and attest that conversion, the audit runs on the quantized version, and the provider may serve the full model 10. That attestation shows where the audited version came from, not that the full model would score the same. Separately, for a public model, Modelwrap lets anyone recompute the served weights' hash from the published weights. For a private model, users see only the hash 11.
- The chain trusts the TEE vendors' attestation keys. With physical access, researchers forged Intel TDX attestations and paired them with relayed H100 attestations, so that a system outside TEE protection passed both checks 13. Sampled recomputation, as in DiFR (R2), avoids hardware trust but needs a trusted copy of the weights and evidence tied to the production serving path 6. Zero-knowledge proofs of inference (R2) keep the weights private without trusting hardware, but proving remains expensive 6.
On this page
Why it matters
Checks on a model's capabilities or safety apply to the model that was checked. Several parties depend on that link:
- RAND's verification framework asks verifiers to confirm that declared inference is declared accurately, so that the prover actually did the claimed deployment 1. It then asks them to confirm that deployed models have the required properties, for example through evaluations at regular intervals 1. Both steps assume the evaluated model is the served model. A low-trust system overview identifies running approved models for prohibited uses as the most difficult violation to defend against, and aims to deploy only approved models 6. The Oxford Martin AI Governance Initiative's verification report includes appendices on model fingerprint attestation and on "device-model mating" with an encrypted model 5.
- The DiFR authors note that providers and their customers increasingly need to verify that inference is performed correctly, without errors or tampering 2. Their Token-DiFR method detected 4-bit quantization of a model with AUC above 0.999 within 300 output tokens 2. In an audit of commercial inference APIs for four Llama models in summer 2024, Gao, Liang and Guestrin found that 11 of 31 endpoints served a different output distribution from the published reference weights 16.
Why it is hard
- Statistical tests on an API's outputs need no cooperation from the provider, but they have limits 16. Gao, Liang and Guestrin's model equality test reached a median statistical power of 77.4% against a range of distortions, using an average of 10 samples per prompt 16. Cai and colleagues find that tests on text outputs are query-intensive and fail against subtle substitutions, and that tests on log probabilities are defeated by inference nondeterminism in production 17.
- Re-running the same inference often gives slightly different results because of benign numerical variation, which makes it hard to tell legitimate variation from substitution 2. In LLM serving a major cause is that kernels are not invariant to batch size, which varies with server load 3. Statistical tests tolerate this noise 2. Alternatively, Cankaya reports that inference can be reproduced bit-exactly across several NVIDIA GPU variants in software, given enough recorded information about the original run 4.
- Recomputation requires the weights, which a provider or state may not reveal 6. One system design keeps weights cryptographically committed and uses them only inside air-gapped auditing environments 6. Zero-knowledge proofs avoid revealing weights at all. zkLLM reports proving one 2,048-token forward pass of a 13-billion-parameter model in under 15 minutes, with proofs under 200 kB 7, but the low-trust overview describes zero-knowledge proof computation as still expensive 6. Trusted execution environments can attest which software and data were loaded 9. PAL*M attests properties on confidential computing hardware (Intel TDX with NVIDIA H100). Its abstract reports under 11% overhead for common operations, but its tables reach 11.24% for MMLU evaluation 8.
- Evidence about a test instance says nothing about production unless it is tied to the serving hardware and time. The low-trust design aims to identify each forward pass uniquely and attribute it to the hardware and time it was processed 6. TEE-based binding relies on the vendor's keys: whoever holds the hardware's attestation key can produce valid reports, and every component that influences inference must be covered by launch measurement 9. A published memory-bus interposition attack, TEE.fail, which needs physical access, extracted a per-CPU Intel attestation key and forged Intel TDX attestations 13. Other attacks forged AMD SEV-SNP attestations, one through a DDR4 interposer (Battering RAM) and one from software alone (RMPocalypse) on platforms without AMD's fix 14 15.