{
  "schema_version": "1.0.0",
  "rubric_version": "1.0",
  "license": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
  "record": {
    "id": "K-0025",
    "slug": "inference-and-training-workloads",
    "title": "Inference and training workloads",
    "aliases": [
      "training vs inference",
      "workload classification"
    ],
    "status": "draft",
    "last_reviewed": "2026-09-23",
    "review_interval_days": 90,
    "steward": null,
    "provenance": {
      "drafted_by": "ai",
      "reviewed_by": []
    },
    "risk_flags": [],
    "flags": [],
    "one_liner": "Training updates a model's weights from data; inference runs fixed weights to produce outputs, and their different resource use underpins several verification methods.",
    "sources": [
      {
        "source": "S-0029",
        "supports": "training steps update weights from data batches; no straightforward way to tell whether an ML chip is running training or an unrelated job",
        "locator": "§5.1; §4"
      },
      {
        "source": "S-0018",
        "supports": "inference yields token-level input-output traffic on front-end links; training uses the back-end fabric",
        "locator": "inference vs training"
      },
      {
        "source": "S-0002",
        "supports": "declared training (1.A.1) and inference (1.A.2) as distinct declared uses",
        "locator": "§3.2"
      },
      {
        "source": "S-0053",
        "supports": "majority of AI compute used for inference; single training run needs far more compute than a single inference; copies of a model run on little compute",
        "locator": "training vs inference discussion"
      },
      {
        "source": "S-0005",
        "supports": "large-scale training links thousands of chips and exchanges gradients; efficient inference on dozens to low hundreds of chips passes only tokens between pods; utilization and power often differ",
        "locator": "Interconnect bandwidth limits; workload classification with high-level chip measures"
      },
      {
        "source": "S-0037",
        "supports": "NVML-telemetry classifier: 98.2% binary accuracy at identifying training across its corpus (9 GPU models); 43–87% against the white-box disguised workloads held out from training, after hardening",
        "locator": "abstract"
      }
    ],
    "related": [
      "K-0021",
      "K-0023",
      "K-0017"
    ],
    "type": "concept",
    "url": "https://trustbutveri.fyi/concepts/inference-and-training-workloads/",
    "source_file": "content/concepts/inference-and-training-workloads.md",
    "flags_all": [
      "ai-drafted"
    ],
    "body_markdown": "Training is the workload that updates a model's weights step by step from batches of data, and inference is the workload that runs a model with fixed weights on inputs to produce outputs such as tokens [[S-0029]] [[S-0018]].\n\nRAND's verification framework treats declared training and declared inference as distinct uses of compute, each to be verified [[S-0002]]. Sastry and colleagues note that most AI compute is used for inference, although a single training run needs far more compute than a single inference, and individual copies of a model can run on relatively little compute [[S-0053]]. Verification designs use the differences in resource use:\n- **Communication.** Large-scale training links thousands of accelerators and exchanges gradients or activations between groups of them, while inference between pods passes only tokens [[S-0005]]; [[M-0014|bandwidth limits]] rely on this gap.\n- **Power and utilization.** Training and inference often differ in accelerator utilization and power draw [[S-0005]], which [[M-0021|workload classification from telemetry and side channels]] uses.\n- **Telemetry.** One classifier using GPU telemetry reports 98.2% binary accuracy at identifying training across its corpus of nine GPU models, falling to 43–87% on the most challenging disguised workloads held out from its training [[S-0037]], as in [[M-0010|on-chip telemetry]].\n\nShavit notes that there is no straightforward way to tell whether an accelerator is running training or an unrelated workload [[S-0029]].",
    "body_text": "Training is the workload that updates a model's weights step by step from batches of data, and inference is the workload that runs a model with fixed weights on inputs to produce outputs such as tokens [S-0029] [S-0018]. RAND's verification framework treats declared training and declared inference as distinct uses of compute, each to be verified [S-0002]. Sastry and colleagues note that most AI compute is used for inference, although a single training run needs far more compute than a single inference, and individual copies of a model can run on relatively little compute [S-0053]. Verification designs use the differences in resource use: - Communication. Large-scale training links thousands of accelerators and exchanges gradients or activations between groups of them, while inference between pods passes only tokens [S-0005]; bandwidth limits rely on this gap. - Power and utilization. Training and inference often differ in accelerator utilization and power draw [S-0005], which workload classification from telemetry and side channels uses. - Telemetry. One classifier using GPU telemetry reports 98.2% binary accuracy at identifying training across its corpus of nine GPU models, falling to 43–87% on the most challenging disguised workloads held out from its training [S-0037], as in on-chip telemetry. Shavit notes that there is no straightforward way to tell whether an accelerator is running training or an unrelated workload [S-0029].",
    "referenced_by": [
      {
        "id": "M-0014",
        "title": "Bandwidth limits and compartmentalization",
        "url": "https://trustbutveri.fyi/mechanisms/bandwidth-limits-and-compartmentalization/"
      },
      {
        "id": "M-0010",
        "title": "On-chip telemetry from timing, memory and performance counters",
        "url": "https://trustbutveri.fyi/mechanisms/on-chip-telemetry/"
      },
      {
        "id": "M-0007",
        "title": "Proofs of useful work and resource exhaustion",
        "url": "https://trustbutveri.fyi/mechanisms/proofs-of-useful-work/"
      },
      {
        "id": "M-0003",
        "title": "Reproducible computation packets",
        "url": "https://trustbutveri.fyi/mechanisms/reproducible-computation-packets/"
      },
      {
        "id": "M-0021",
        "title": "Workload classification from telemetry and side channels",
        "url": "https://trustbutveri.fyi/mechanisms/workload-classification-from-telemetry/"
      },
      {
        "id": "I-0011",
        "title": "AI 2040 inference-only verification stack",
        "url": "https://trustbutveri.fyi/implementations/ai-2040-inference-only-verification-plan/"
      },
      {
        "id": "I-0008",
        "title": "SASH confidential network logger",
        "url": "https://trustbutveri.fyi/implementations/sash-confidential-network-logger/"
      },
      {
        "id": "C-0003",
        "title": "Declared hardware is idle or shut down",
        "url": "https://trustbutveri.fyi/claims/declared-hardware-is-idle/"
      },
      {
        "id": "C-0004",
        "title": "This compute runs inference, not training",
        "url": "https://trustbutveri.fyi/claims/inference-not-training/"
      },
      {
        "id": "K-0017",
        "title": "Compartmentalization",
        "url": "https://trustbutveri.fyi/concepts/compartmentalization/"
      },
      {
        "id": "K-0023",
        "title": "FLOP accounting",
        "url": "https://trustbutveri.fyi/concepts/flop-accounting/"
      },
      {
        "id": "K-0021",
        "title": "Interconnect bandwidth",
        "url": "https://trustbutveri.fyi/concepts/interconnect-bandwidth/"
      }
    ]
  }
}