AI Verification LandscapeSearch

Glossary

Inference and training workloads

Training updates a model's weights from data; inference runs fixed weights to produce outputs, and their different resource use underpins several verification methods.

Training is the workload that updates a model's weights step by step from batches of data, and inference is the workload that runs a model with fixed weights on inputs to produce outputs such as tokens 1 2.

RAND's verification framework treats declared training and declared inference as distinct uses of compute, each to be verified 3. Sastry and colleagues note that most AI compute is used for inference, although a single training run needs far more compute than a single inference, and individual copies of a model can run on relatively little compute 4. Verification designs use the differences in resource use:

  • Communication. Large-scale training links thousands of accelerators and exchanges gradients or activations between groups of them, while inference between pods passes only tokens 5; bandwidth limits rely on this gap.
  • Power and utilization. Training and inference often differ in accelerator utilization and power draw 5, which workload classification from telemetry and side channels uses.
  • Telemetry. One classifier using GPU telemetry reports 98.2% binary accuracy at identifying training across its corpus of nine GPU models, falling to 43–87% on the most challenging disguised workloads held out from its training 6, as in on-chip telemetry.

Shavit notes that there is no straightforward way to tell whether an accelerator is running training or an unrelated workload 1.

Related

Used in

Sources

  1. BY. Shavit (2023). What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring. arXiv. Source recordSupports: training steps update weights from data batches; no straightforward way to tell whether an ML chip is running training or an unrelated job · §5.1; §4
  2. BN. Cankaya (2026). A System Overview for Near-Term, Low-Trust AI Compute Verification. Machine Intelligence Research Institute. Source recordSupports: inference yields token-level input-output traffic on front-end links; training uses the back-end fabric · inference vs training
  3. BM. Baker et al. (2025). Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment. RAND Corporation. Source recordSupports: declared training (1.A.1) and inference (1.A.2) as distinct declared uses · §3.2
  4. BG. Sastry et al. (2024). Computing Power and the Governance of Artificial Intelligence. arXiv. Source recordSupports: majority of AI compute used for inference; single training run needs far more compute than a single inference; copies of a model run on little compute · training vs inference discussion
  5. BA. Scher & L. Thiergart (2025). Mechanisms to Verify International Agreements About AI Development. arXiv. Source recordSupports: large-scale training links thousands of chips and exchanges gradients; efficient inference on dozens to low hundreds of chips passes only tokens between pods; utilization and power often differ · Interconnect bandwidth limits; workload classification with high-level chip measures
  6. BR. Rahman & S. Tajdari (2026). Detecting Hidden ML Training With Zero-Overhead Telemetry. ICML 2026 Workshop on Technical AI Governance Research. Source recordSupports: NVML-telemetry classifier: 98.2% binary accuracy at identifying training across its corpus (9 GPU models); 43–87% against the white-box disguised workloads held out from training, after hardening · abstract