Sampled inference recomputation

How it works

The provider records its inference traffic, and a verifier re-runs a random sample of it. The provider's serving system logs each request and response with the exact model release, code version and random seed used ¹. Proposals for verifying international agreements, such as the AI 2040 plan, use passive network taps on the front-end network to copy all input and output traffic to a recomputation server ⁶. A verification server then re-runs a random subset of the logged inferences with a trusted copy of the model and code ¹.

Checking is cheaper than generating ¹. The verifier feeds the prompt and the claimed output through the model as one sequence. This gives the model's predictions for every position at once ². Generation must produce tokens one at a time, which typically uses hardware 3–5 times less efficiently than the single pass ¹.

The comparison must tolerate numerical noise. Re-running the same inference often gives slightly different results, because floating-point arithmetic is not associative ^{2 3}. Published schemes handle this in three ways:

• TOPLOC compares activation fingerprints. The provider records a compact encoding of the largest last-layer activations. The verifier accepts if its recomputed values differ by less than preset thresholds ^{3 7}.
• Token-DiFR, part of DiFR, checks tokens under a shared sampling seed. The verifier scores how far each claimed token departs from the token the reference model would have chosen ².
• Exact replay needs no tolerance. Recomputation can match bit for bit if the verifier knows the hardware model, deployed weights, parallelism layout, software versions and batch size ^{4 5}.

A modest random sample gives statistical evidence about the whole recorded stream ^{4 6}. If 0.1% of outputs are false and every check catches a false output, 3,000 random checks find at least one with 95.03% probability ⁴.

What it establishes

Sampled recomputation can establish two things:

• It can show that recorded outputs are consistent with the declared model and settings. In DiFR's experiments, recomputation detected 4-bit quantization, a quantized key-value cache, a wrong temperature, a wrong seed and simulated sampling bugs ². TOPLOC detected changes to the model, the prompt and the numerical precision ⁷.
• It can limit how much information a compromised server can hide in its responses ¹.

It cannot establish three things:

• It checks only recorded workloads, so it cannot show that no other workloads ran unrecorded ³. Rinberg et al. say their weight-exfiltration scheme cannot be a defence on its own ¹.
• Schemes that tolerate small differences can bound an adversary's covert bandwidth but cannot close it ⁵.
• Checks based only on activations cannot verify that tokens were sampled correctly ².

Threat model

The adversary controls the inference server ¹. Rinberg et al.'s exfiltration study makes these trust assumptions, among others ¹:

• There are no side channels.
• The verification code is trusted.
• The log is secure and append-only.
• Input and output messages are paired.
• The adversary can neither see nor manipulate the sample selection, nor choose or bias generation seeds.

The threat model lets external adversaries fully control user queries, but the published bound was measured on benign prompts from a public chat dataset ¹. An independent study found that an adversary who controls the prompts can widen it ¹⁶.

The prover must cooperate. In Amodo's example schemes for AI agreements, it must let the verifier install a recomputation server in its data centre ³. The verifier must hold the model weights and know the provider's sampling procedure ². Tolerance thresholds come from calibration runs on trusted hardware ². One low-trust system design treats physical security of the installed devices as a prerequisite ⁴.