{
  "schema_version": "1.0.0",
  "rubric_version": "1.0",
  "license": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
  "record": {
    "id": "K-0008",
    "slug": "numerical-nondeterminism",
    "title": "Numerical nondeterminism",
    "aliases": [
      "floating-point nondeterminism",
      "non-reproducibility"
    ],
    "status": "draft",
    "last_reviewed": "2026-09-23",
    "review_interval_days": 90,
    "steward": null,
    "provenance": {
      "drafted_by": "ai",
      "reviewed_by": []
    },
    "risk_flags": [],
    "flags": [],
    "one_liner": "Differences between runs, or between machines, in the results of the same AI computation, because floating-point rounding depends on the order of operations.",
    "sources": [
      {
        "source": "S-1010",
        "supports": "GPU non-determinism arises from unspecified details including rounding strategy, subnormal numbers and accumulation order, since floating-point arithmetic is not associative; results differ between GPU architectures",
        "locator": "abstract; §1"
      },
      {
        "source": "S-0016",
        "supports": "re-running the same inference often gives different results due to benign numerical noise; comparison against a trusted reference conditioned on the same sampling seed",
        "locator": "abstract"
      },
      {
        "source": "S-1009",
        "supports": "floating-point non-associativity; LLM forward pass run-to-run deterministic; lack of batch invariance with load-dependent batch size as the main cause of nondeterminism in LLM inference endpoints; batch-invariant kernels",
        "locator": "sections on non-associativity, the concurrency hypothesis and batch invariance"
      },
      {
        "source": "S-0018",
        "supports": "metadata needed for bit-exact replay (hardware SKU, quantization, parallelism, kernels, batch size)",
        "locator": "recomputation section"
      },
      {
        "source": "S-0029",
        "supports": "accept a recomputed checkpoint within a small distance of the reported one",
        "locator": "§5.1"
      },
      {
        "source": "S-0020",
        "supports": "software emulation predicting every bit of transformer forward passes across NVIDIA GPU architectures, validated on dense blocks on A100, L40, L40S and H100",
        "locator": "abstract; evaluation"
      }
    ],
    "related": [
      "K-0009"
    ],
    "type": "concept",
    "url": "https://trustbutveri.fyi/concepts/numerical-nondeterminism/",
    "source_file": "content/concepts/numerical-nondeterminism.md",
    "flags_all": [
      "ai-drafted"
    ],
    "body_markdown": "Numerical nondeterminism is variation in the results of the same computation on the same inputs, across repeated runs or across hardware and software setups, that comes from floating-point arithmetic rather than from intended randomness such as sampling [[S-1010]] [[S-0016]].\n\nFloating-point addition is not associative, so a rounded sum depends on the order in which its terms are accumulated [[S-1009]] [[S-1010]]. GPUs leave that order, their rounding strategy and their handling of subnormal numbers unspecified, and the same matrix multiplication can give different results on different GPU architectures [[S-1010]]. On a single machine, a Thinking Machines post finds the LLM forward pass run-to-run deterministic for a fixed batch, and traces the variation users see to kernels whose results change with batch size, which depends on server load [[S-1009]]. Exact replay can therefore need the original hardware model, quantization, parallelism layout, kernels and batch size [[S-0018]]. For a verifier, this makes legitimate variation hard to tell from real problems [[S-0016]], and verification designs respond in two ways:\n- **Tolerate it.** [[M-0001|Sampled inference recomputation]] compares outputs with a trusted reference that uses the same sampling seed [[S-0016]], and [[M-0006|proof-of-learning]] accepts a recomputed checkpoint within a small distance of the reported one [[S-0029]].\n- **Remove it.** [[M-0002|Deterministic and bit-exact inference]] uses batch-invariant kernels [[S-1009]] or software emulation that predicts, bit for bit, the outputs of dense transformer blocks on four NVIDIA GPU models [[S-0020]].",
    "body_text": "Numerical nondeterminism is variation in the results of the same computation on the same inputs, across repeated runs or across hardware and software setups, that comes from floating-point arithmetic rather than from intended randomness such as sampling [S-1010] [S-0016]. Floating-point addition is not associative, so a rounded sum depends on the order in which its terms are accumulated [S-1009] [S-1010]. GPUs leave that order, their rounding strategy and their handling of subnormal numbers unspecified, and the same matrix multiplication can give different results on different GPU architectures [S-1010]. On a single machine, a Thinking Machines post finds the LLM forward pass run-to-run deterministic for a fixed batch, and traces the variation users see to kernels whose results change with batch size, which depends on server load [S-1009]. Exact replay can therefore need the original hardware model, quantization, parallelism layout, kernels and batch size [S-0018]. For a verifier, this makes legitimate variation hard to tell from real problems [S-0016], and verification designs respond in two ways: - Tolerate it. Sampled inference recomputation compares outputs with a trusted reference that uses the same sampling seed [S-0016], and proof-of-learning accepts a recomputed checkpoint within a small distance of the reported one [S-0029]. - Remove it. Deterministic and bit-exact inference uses batch-invariant kernels [S-1009] or software emulation that predicts, bit for bit, the outputs of dense transformer blocks on four NVIDIA GPU models [S-0020].",
    "referenced_by": [
      {
        "id": "M-0024",
        "title": "Bounding unexplained information in outputs",
        "url": "https://trustbutveri.fyi/mechanisms/bounding-unexplained-information/"
      },
      {
        "id": "M-0002",
        "title": "Deterministic and bit-exact inference",
        "url": "https://trustbutveri.fyi/mechanisms/deterministic-inference/"
      },
      {
        "id": "M-0012",
        "title": "Model identity attestation",
        "url": "https://trustbutveri.fyi/mechanisms/model-identity-attestation/"
      },
      {
        "id": "M-0013",
        "title": "Network taps and certifiers",
        "url": "https://trustbutveri.fyi/mechanisms/network-taps-and-certifiers/"
      },
      {
        "id": "M-0006",
        "title": "Proof-of-learning and training-transcript verification",
        "url": "https://trustbutveri.fyi/mechanisms/proof-of-learning/"
      },
      {
        "id": "M-0001",
        "title": "Sampled inference recomputation",
        "url": "https://trustbutveri.fyi/mechanisms/sampled-inference-recomputation/"
      },
      {
        "id": "M-0005",
        "title": "Zero-knowledge proofs of training constraints",
        "url": "https://trustbutveri.fyi/mechanisms/zk-proofs-of-training-constraints/"
      },
      {
        "id": "I-0002",
        "title": "DiFR (Divergence From Reference)",
        "url": "https://trustbutveri.fyi/implementations/difr/"
      },
      {
        "id": "I-0004",
        "title": "Pearl proof-of-useful-work blockchain",
        "url": "https://trustbutveri.fyi/implementations/pearl-proof-of-useful-work/"
      },
      {
        "id": "I-0001",
        "title": "TOPLOC",
        "url": "https://trustbutveri.fyi/implementations/toploc/"
      },
      {
        "id": "C-0005",
        "title": "The declared model is the one being served",
        "url": "https://trustbutveri.fyi/claims/declared-model-is-served/"
      },
      {
        "id": "C-0004",
        "title": "This compute runs inference, not training",
        "url": "https://trustbutveri.fyi/claims/inference-not-training/"
      },
      {
        "id": "C-0007",
        "title": "A training run stayed within declared limits",
        "url": "https://trustbutveri.fyi/claims/training-within-declared-limits/"
      },
      {
        "id": "K-0009",
        "title": "Recomputation",
        "url": "https://trustbutveri.fyi/concepts/recomputation/"
      }
    ]
  }
}