Low-trust AI compute verification system overview

What it is

Naci Cankaya's overview, from the Machine Intelligence Research Institute's Technical Governance Team, is a working draft ¹. It describes the author's current best idea for a privacy-preserving, retrofittable AI compute verification system, meant to build confidence in an arms-control-style AI agreement ¹. It offers example rules the system should support: monitorable chains of thought, cryptographic commitment of checkpoints, whitelisting of models before deployment, and a blacklist of illicit uses ¹.

The author argues that checking aggregate properties, such as total training FLOPs, is not enough, and sets collision-resistant evidence capture of individual workloads as a requirement ¹. He assumes the worst case on both sides: the prover may covertly break the agreement's rules, and the verifier may try to exfiltrate the prover's secrets ¹.

How it works

The design separates capturing evidence from evaluating it ¹.

Evidence capture

• Network taps. Taps on the monitored links hash traffic in groups of packets at line rate ¹. See Network taps and certifiers.
• Shared hash stream. The hashes leave over a split, sanitized fibre, so prover and verifier see identical copies. The prover recomputes the hashes from its own copy of the traffic, confirming that the tap sent nothing but legitimate commitments ¹.
• Prover-held data. Beyond hashes, no information is shared with the verifier; the prover keeps the plaintext and a manifest ¹.
• Memory challenges and wiping. These are an optional second form of evidence ¹. See Timed challenge-response and memory-occupation challenges and Memory wiping and proofs of secure erasure.

A companion preprint specifies a secure gateway device for the taps that needs no processor trusted by both parties ².

Evidence evaluation

1. The verifier issues a challenge over a one-way link enforced by a data diode ¹.
2. The prover supplies the matching data, manifest entries and replay metadata ¹.
3. Each party's trusted computing base checks the hashes, re-runs the computation under the declared metadata and screens it for compliance ¹. See Deterministic and bit-exact inference.
4. The trusted computing bases cross-check their verdicts, and nothing is released unless they agree; by default, disclosure is minimal ¹.

A missing or mismatched record fails the challenge, while a failed replay or a disagreement triggers investigation ¹. The evaluation hardware is at least three distinct trusted computing bases from different supply chains, trusted respectively by the prover, the verifier and a third party, each inside its own air-gap in the prover's data centre ¹. Zero-knowledge proofs are a tentative "plan B" for evaluation, if they mature ¹. See Zero-knowledge proofs of inference.

Subsystems

The design avoids the need for mutually trusted silicon with these building blocks ¹:

• Analog data movement control. Fused fibre splitters duplicate traffic, optical isolators act as data diodes, and enclosures protect the equipment ¹.
• Controlled ingress. A one-way path carries the verifier's inputs into the auditing environment ¹.
• Output cross-checks. The prover clones the output of each verifier device with a fibre splitter, to confirm it sends nothing but the correctly computed result ¹.
• Sanitized egress. The prover re-transmits verifier outputs at batched or randomized times, blocking optical and timing side channels ¹.
• Secure performance offshoring. Untrusted fast hardware re-runs workloads, while small trusted devices compare the results ¹.

Side-channel defence and resource accounting support the rest of the system ¹. See Side-channel suppression for isolated facilities.

Evidence

• Status. The overview sets out a design and open research questions ¹. Its companion tap preprint states that empirical validation is still required ².
• Prior work it builds on. Verde obtained bitwise-identical inference results across several NVIDIA GPUs by controlling the order of floating-point operations ¹. The TrustGuard sentry, which re-executes instructions, was prototyped on an FPGA ¹.
• Taps. Amodo Design is investigating passive optical splitting at 53–112 GBaud ¹. The companion preprint expects a demonstration gateway to cost roughly as much to develop as a small team of engineers for a few months ².
• Cost target. The author expects acceptance to depend on retrofit costs below 10% of the monitored hardware, ideally below 1% ¹.

Limitations

• Attribution. A technical mismatch does not show whether it came from evasion, a random bit flip or faulty evaluation software ¹.
• Fault leakage. A malicious device can leak one bit per deliberately wrong output, so the design needs a fault budget ¹.
• Inspector agents. Screening agents must resist prompt injection ¹.
• Physical security. Securing every monitored data centre against covert communication is challenging ¹.
• Zero-knowledge option. Proofs work over integers, while accelerated inference accumulates floating-point rounding errors ¹.
• Threat model. The author calls the threat model section under-developed ¹.