Bounding unexplained information in outputs

How it works

The idea is to measure the information that leaves a facility against what its declared work would produce ¹. Petrie and Mühlhäuser propose verifying AI compute agreements by bounding the amount of unexplained information in network outputs, without either party trusting the other's hardware ¹.

• Isolation and commitment. The prover's compute nodes are isolated, and all traffic passes through a verifier interlock that records cryptographic commitments of the encrypted traffic ¹.
• Prediction challenges. From time to time the verifier challenges the prover to show that randomly selected outputs can be predicted by a policy-compliant computation on the measured inputs ¹.
• Principle. The better compliant computation predicts the outputs, the less bandwidth is left for exporting the results of undeclared workloads ¹.
• Confidentiality. The prover can recompute inside a verifier-controlled enclosure, the verifier can recompute inside a prover-controlled enclosure, or zero-knowledge proofs can be used ¹.

For language-model inference, Rinberg et al. make this concrete. Honest inference follows a well-defined generative process, so hiding data in responses must make them deviate from expected model behaviour ². A verification server re-runs sampled requests with the trusted model and the shared random seed, scores how plausible each emitted token is, and bounds the rate at which hidden bits can pass unnoticed ². See Sampled inference recomputation.

A related line of work uses compression ^{5 4}. One proposal compresses each response with a weaker reference model, using arithmetic coding, so that only bits the reference model did not predict count against an upload limit ⁵. Rinberg and co-authors note that limiting egress rests on a simple fact: if only 10 GB leaves a data centre, no attacker can steal more than 10 GB ⁴. Language-model text compresses far better than model weights, so a facility can apply stricter egress limits ⁴.

What it establishes

It can put an upper bound on the covert information in checked outputs, which slows exfiltration rather than stopping it ². Under benign prompt traffic, Rinberg et al. report that the exfiltratable fraction fell to under 0.5%, not zero ².

It does not establish:

• That explained outputs are harmless. It does not limit information that the declared computation itself explains ⁴. Compression research notes that someone with inference access can extract more proprietary information per bit than naive transmission allows ⁴.
• That no smaller model is being prepared. One upload-limit proposal lists running a hidden distillation job inside the data centre, to produce a smaller model that is easier to move, as a residual route ⁵.
• Anything about other channels. The inference scheme treats side channels as out of scope ².

Threat model

• Facility-level design. Neither party trusts the other's hardware, and the prover's compute can be isolated behind the interlock ¹.
• Inference scheme. The adversary controls the server's outputs ². The scheme also assumes trusted verification code, secure logging, private sampling that the adversary cannot manipulate, and randomness that is trusted and non-malleable ². An independent study notes that the published bound assumes a passive attacker sending benign prompts ³.
• Upload limits. The proposal assumes strong physical security ⁵. It also assumes that a lab's most powerful model will not generate more than one or two orders of magnitude more tokens than it saw in training ⁵.

Evidence

• Inference outputs. Rinberg et al. tested models from 3 to 30 billion parameters, including two mixture-of-experts models, and published code ². On the 30-billion-parameter mixture-of-experts model, under benign prompt traffic, they report under 0.5% exfiltratable information at a false-positive rate below 0.01%, a slowdown of more than 200 times ².
• Independent attack. Kezins tested six instruction-tuned models of 1 to 32 billion parameters over three seeds ³. Engineered prompts roughly doubled leakage per token and reduced the slowdown to 60–118 times ³.
• Compressibility. A question-asking protocol compressed responses to ratios of 0.0006 to 0.004 while recovering part of the gap between a small and a large model ⁴.
• Facility-level architecture. Published as protocol details, attack vectors and prototyping plans ¹.

Limitations

• Entropy inflation. An attacker who controls prompts widens the set of tokens the verifier tolerates ³. Kezins argues that designs built on the same unexplained-information bound inherit this weakness, and recommends calibrating tolerances against local token entropy ³.
• Noise tolerance. The covert channel lives in the tolerance allowed for GPU nondeterminism ³. Bit-exact replay needs the hardware model, weights, parallelism layout, software versions and batch size ⁶; see Deterministic and bit-exact inference.
• Residual physical bandwidth. One design argues that suppressing covert physical bandwidth below kilobits per second is much more achievable than aiming for zero ⁶. See Side-channel suppression for isolated facilities and Bandwidth limits and compartmentalization.
• Uncertain volumes. The upload-limit author is uncertain about the ratio of generated tokens to model parameters ⁵. He estimates about a 25% chance that the approach can cheaply ensure that weights cannot be taken without bypassing physical security ⁵.