Mechanism · Training-transcript verification (proof-of-learning)
Evidence & limits
On this page
R2Demonstrated for checking from its transcript that a training run followed declared rules
R2 through later peer-reviewed work by Choi et al. and Srivastava et al. The original verification rule is broken, and no independent attack on the later defences has been published.
Assessed use: checking from its transcript that a training run followed declared rules
Rubric assessment
- R1 met: Jia et al. define proof-of-learning and its security goal 1, and Shavit sets out training-transcript verification for rules on large training runs 5.
- R2 met through later work, not through the original scheme. The original authors demonstrated spoofs that pass verification across configurations at a fraction of the cost of earlier spoofing strategies 4. An earlier independent attack reported spoofs cheaper than honest training 3, which the original authors dispute 4. These breaks invalidate the original demonstration, including the public code, which implements that rule 2. Choi et al. publish peer-reviewed end-to-end experiments against a stated set of spoofing attacks, training GPT-2 (124M) and testing published Pythia checkpoints up to 1B on GPUs 6. The paper links no code; the rubric does not require code for results described in enough detail to repeat. Srivastava et al. replicate ResNet-50 training and GPT-2 fine-tuning bit for bit on three NVIDIA GPU types, against a trainer that may poison data, and release their code 7. The open critical flaws concern the original rule, not the evidence the level rests on.
- R3 not met: Verde realises this mechanism for delegated training jobs, but Gensyn states that its public runtime is limited to LLM inference 9, so its production use does not count for training verification. As of September 2026 no third party is known to rely on training-transcript checks, and no production-grade implementation has been published.
- R4 not met: no independent attack on the post-2023 defences has been published.
- Use by a party other than the developer, or a production-grade implementation, at realistic training scale.
- Verification rules with formal robustness arguments, as Fang et al. argue are needed, or an independent red-team of the post-2023 defences.
Assessed 2026-09-25 against rubric v1.1.
Evidence
- Jia et al. They evaluate PoL with ResNet-20 and ResNet-50 on CIFAR-10 and CIFAR-100 1. Their code for creating and verifying proofs is public 2.
- Choi et al. They train GPT-2 (124M) on about 8.8 billion OpenWebText tokens, and apply their tests to EleutherAI's published Pythia checkpoints of 70M, 410M and 1B parameters, using four A40 GPUs 6. They show detection of attacks from the PoL literature, including gluing, interpolation and data addition or subtraction 6. Sampling 1% of the points in each segment, they estimate that their new memorisation and data-order tests cost "just 1.3% of the original cost of training", assuming inference is three times cheaper than training, in addition to retraining selected segments 6.
- Srivastava et al. They replicate ResNet-50 (23M) training on CIFAR-10 and GPT-2 (117M) fine-tuning exactly at FP32 across NVIDIA A40, Titan XP and RTX 2080 Ti GPUs. Their prototype adds 20 to 40% to the trainer's time, and the auditor takes 1.3 to 1.7 times as long as training. Their code is public 7.
- Verde. Gensyn's Verde narrows disputes between compute providers over a delegated training job to one step, then to one operation, which a referee re-runs 8. The paper reports that its reproducible operators added 126% to LoRA fine-tuning time for Llama-8B on an A100 8. Gensyn states that its public runtime is limited to LLM inference 9.
- Shavit. His framework is a proposal, with estimates of how many chips inspectors would need to sample 5.
Limitations
Independent attack. Zhang et al. of Zhejiang University showed that adversarial-example-style perturbations let an attacker generate a passing proof "with significantly less cost" than honest training 3. Fang et al. dispute this. They state that the attack assumes the adversary chooses the checkpoint interval, which the verifier should set, and that a small interval prevents it 4.
Attack by the original authors. Fang et al., whose authors include all seven original PoL authors, present cheaper spoofs that work across PoL configurations. They find that "current PoL verification is not robust to adversaries", and show that the assumptions needed for robust verification reduce to open problems in learning theory 4.
Gaps in later defences. Choi et al. defend against several known attacks, but note that small-scale data changes and masked hyperparameters remain open 6. In Srivastava et al.'s scheme, the auditor re-runs the whole run, and all parties must trust it. A trainer that knows the auditor's GPU can choose among models that differ only in rounding decisions at the few steps where the auditor is close to a rounding boundary. Distributed training is left to future work 7.
Cost. Storage is a burden: checkpoints may each take terabytes 5. Shavit identifies cheaper alternatives to retraining-based verification as future work 5.
Alternatives. Zhang et al. suggest proofs built on verifiable computation as a countermeasure 3. Zero-knowledge proofs of training take that route, proving each training step cryptographically 10.
Known flaws
- Adversarial-example spoofs pass verification at lower cost than training
- Structurally correct spoofs exploit tolerance thresholds and sampled checks
- No provably robust verification without better optimisation theory
- Rule-compliance transcripts face a stronger adversary
- Small data changes and masked hyperparameters may go undetected
Blockers
Transcripts are large: weight checkpoints may each require terabytes 5.
The verifier must reproduce training segments, which may be infeasible if the prover uses specialised or proprietary hardware 6.
The noise tolerance needed for honest reproduction is what structurally correct spoofs exploit 4.
Tying transcripts to real chips needs on-chip weight-snapshot logging, chip inspections and a trusted chip-owner directory 5.