Numerical nondeterminism
Differences between runs, or between machines, in the results of the same AI computation, because floating-point rounding depends on the order of operations.
Numerical nondeterminism is variation in the results of the same computation on the same inputs, across repeated runs or across hardware and software setups, that comes from floating-point arithmetic rather than from intended randomness such as sampling 1 2.
Floating-point addition is not associative, so a rounded sum depends on the order in which its terms are accumulated 3 1. GPUs leave that order, their rounding strategy and their handling of subnormal numbers unspecified, and the same matrix multiplication can give different results on different GPU architectures 1. On a single machine, a Thinking Machines post finds the LLM forward pass run-to-run deterministic for a fixed batch, and traces the variation users see to kernels whose results change with batch size, which depends on server load 3. Exact replay can therefore need the original hardware model, quantization, parallelism layout, kernels and batch size 4. For a verifier, this makes legitimate variation hard to tell from real problems 2, and verification designs respond in two ways:
- Tolerate it. Sampled inference recomputation compares outputs with a trusted reference that uses the same sampling seed 2, and proof-of-learning accepts a recomputed checkpoint within a small distance of the reported one 5.
- Remove it. Deterministic and bit-exact inference uses batch-invariant kernels 3 or software emulation that predicts, bit for bit, the outputs of dense transformer blocks on four NVIDIA GPU models 6.
Related
Used in
- R2Bounding unexplained information in outputs
- R2Deterministic and bit-exact inference
- R2Model identity attestation⚠
- R1Network taps and certifiers
- R2Proof-of-learning and training-transcript verification⚠
- R2Sampled inference recomputation
- R2Zero-knowledge proofs of training constraints
- R2DiFR (Divergence From Reference)
- R2Pearl proof-of-useful-work blockchain
- R2TOPLOC
- The declared model is the one being served
- This compute runs inference, not training
- A training run stayed within declared limits
Sources
- AE. Badash et al. (2026). Hawkeye: Reproducing GPU-Level Non-Determinism. Proceedings of Machine Learning and Systems 8 (MLSys 2026). Source recordSupports: GPU non-determinism arises from unspecified details including rounding strategy, subnormal numbers and accumulation order, since floating-point arithmetic is not associative; results differ between GPU architectures · abstract; §1
- BA. Karvonen et al. (2025). DiFR: Inference Verification Despite Nondeterminism. arXiv. Source recordSupports: re-running the same inference often gives different results due to benign numerical noise; comparison against a trusted reference conditioned on the same sampling seed · abstract
- CH. He & Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism. Source recordSupports: floating-point non-associativity; LLM forward pass run-to-run deterministic; lack of batch invariance with load-dependent batch size as the main cause of nondeterminism in LLM inference endpoints; batch-invariant kernels · sections on non-associativity, the concurrency hypothesis and batch invariance
- BN. Cankaya (2026). A System Overview for Near-Term, Low-Trust AI Compute Verification. Machine Intelligence Research Institute. Source recordSupports: metadata needed for bit-exact replay (hardware SKU, quantization, parallelism, kernels, batch size) · recomputation section
- BY. Shavit (2023). What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring. arXiv. Source recordSupports: accept a recomputed checkpoint within a small distance of the reported one · §5.1
- BN. Cankaya (2026). Bit-Exact AI Inference Verification Without Performance Tradeoffs. ICML 2026 Workshop on Technical AI Governance Research. Source recordSupports: software emulation predicting every bit of transformer forward passes across NVIDIA GPU architectures, validated on dense blocks on A100, L40, L40S and H100 · abstract; evaluation