{
  "schema_version": "1.0.0",
  "rubric_version": "1.0",
  "license": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
  "record": {
    "id": "M-0002",
    "slug": "deterministic-inference",
    "title": "Deterministic and bit-exact inference",
    "aliases": [
      "Bit-exact inference",
      "Batch-invariant inference",
      "Reproducible inference"
    ],
    "status": "draft",
    "last_reviewed": "2026-09-23",
    "review_interval_days": 90,
    "steward": null,
    "provenance": {
      "drafted_by": "ai",
      "reviewed_by": []
    },
    "risk_flags": [],
    "flags": [],
    "one_liner": "Making model inference reproducible bit for bit, so that a verifier's re-run must match the provider's output exactly rather than approximately.",
    "summary": "Re-running the same AI model on the same input often gives slightly different numbers. Floating-point results depend on the order of operations, which shifts with batch size, hardware and software. This noise forces recomputation checks to accept approximate matches, which a cheating provider could exploit. Deterministic inference removes the noise in one of two ways: kernels whose results do not depend on batch size, or recording enough about the hardware and software for a verifier to reproduce every rounding error. vLLM and SGLang offer batch-invariant or deterministic modes, and a published software emulator predicts, bit for bit, the outputs of dense transformer blocks on four NVIDIA GPU models. The obstacles are the throughput cost of batch-invariant kernels, gaps in the emulator's coverage (mixture-of-experts models, one proprietary kernel family, non-NVIDIA GPUs), no independent security evaluation, and the provider's need to disclose its full configuration.",
    "technical": "Two routes lead to exact results.\n\n- **Invariance.** Kernels fix the reduction order for each output element regardless of batch size. Thinking Machines made RMSNorm, matrix multiplication and attention batch-invariant, the last with a fixed split size for the key-value dimension rather than a fixed number of splits [[S-1009]]. vLLM exposes this behind VLLM_BATCH_INVARIANT=1 on NVIDIA GPUs of compute capability 8.0 or higher, in beta [[S-1013]]. SGLang integrated batch-invariant attention for its FlashInfer, FlashAttention 3 and Triton backends [[S-1012]]. LLM-42 instead decodes on a non-deterministic fast path and replays candidate tokens under a fixed-shape reduction schedule, rolling back any that are inconsistent [[S-1014]].\n- **Record and replay.** Stock engines are deterministic but not invariant. Outputs are bitwise reproducible if the verifier knows the hardware model, the exact deployed weights, the parallelism topology (separately for prefill and decode), software versions including custom kernels, and the batch size of each forward pass [[S-0018]] [[S-0020]]. Of these, only batch size changes during serving, and it costs one extra integer per forward pass to record [[S-0020]]. A software emulator reproduces the rounding of other GPU models by modelling tensor-core accumulation and kernel-specific reduction trees [[S-0020]]. Hawkeye reproduces tensor-core matrix multiplication exactly on a CPU for Ampere, Hopper and Ada Lovelace GPUs in FP16, BF16 and FP8 [[S-1010]].\n\nWith exact replay, verification is pass/fail, and the chance of catching at least one false output in k samples is 1 − (1 − p)^k for a false-output rate p [[S-0020]].",
    "category": "cryptographic-computational",
    "secondary_categories": [],
    "verifies": [
      {
        "claim": "C-0005",
        "role": "primary",
        "note": "Enables exact-match recomputation checks that the declared model, weights and software setup produced the outputs."
      },
      {
        "claim": "C-0004",
        "role": "supporting",
        "note": "Bit-exact recomputation of declared inference removes the tolerance an operator could hide other work in (S-0020)."
      },
      {
        "claim": "C-0009",
        "role": "supporting",
        "note": "Removes the tolerance margin that steganographic exfiltration could use (S-0020)."
      },
      {
        "claim": "C-0010",
        "role": "supporting",
        "note": "Unreported batch elements alter the numerics, so covert computation inside batches becomes detectable (S-0020)."
      }
    ],
    "threat_model": "adversarial",
    "adversarial_evaluation": "analysis",
    "hardware_requirement": "none",
    "prover_cooperation": "required",
    "confidentiality": "partial",
    "depends_on": [],
    "readiness": {
      "assessment": true,
      "level": "R2",
      "rubric_version": "1.0",
      "rationale": "R2: public code reproduces dense LLM blocks bit for bit on realistic GPUs against a stated adversary, but no third party relies on exact replay and no independent evaluation exists.\n\n- **R1** met: the verification claim, a covert-adversary threat model and the information a verifier needs are published [[S-0020]].\n- **R2** met: public code predicts dense LLM blocks bit for bit on A100, L40, L40S and H100 GPUs running unmodified vLLM and Hugging Face engines, against a stated adversary [[S-0020]]. Batch-invariant modes are public in vLLM (beta) and SGLang [[S-1013]] [[S-1012]], and batch invariance was shown on a 235-billion-parameter model [[S-1009]]. Exact CPU reproduction of GPU matrix multiplication is peer-reviewed [[S-1010]]. The only Implementation record for this mechanism, [[I-0012]], is a proposed architecture assessed R1, so the level rests on these systems directly.\n- **R3** not met: determinism modes are available in production engines, and one model developer reports using end-to-end deterministic kernels [[S-1011]], but no party other than a developer is documented relying on exact replay for a verification decision. As of September 2026 no independent audit, red-team or peer-reviewed security analysis of bit-exact verification has been published.\n\nConfidence is medium because the verification-specific evidence is one single-author preprint covering dense model blocks.",
      "evidence": [
        "S-0020",
        "S-1009",
        "S-1010",
        "S-1012",
        "S-1013"
      ],
      "next_level_gaps": [
        "Reliance by a party other than the developer on exact-match recomputation for a verification decision, or a production-grade verification stack built on it.",
        "An independent public evaluation (audit, red-team or peer-reviewed security analysis) of bit-exact verification."
      ],
      "confidence": "medium",
      "assessed_by": [
        "ai-draft"
      ],
      "assessed_on": "2026-09-23",
      "status": "current",
      "dispute": null
    },
    "flaws": [
      {
        "assessment": true,
        "title": "Some kernels remain genuinely nondeterministic",
        "kind": "open-question",
        "severity": "minor",
        "status": "open",
        "description": "The bit-exact work separates kernels that are deterministic but not batch-invariant from truly nondeterministic ones that use atomic functions. Some integer de-quantization kernels use atomic additions and remain nondeterministic, so exact replay needs backends that avoid them [[S-0020]].",
        "sources": [
          "S-0020"
        ],
        "response": null
      },
      {
        "assessment": true,
        "title": "Cross-hardware replay relies on reverse-engineered, closed behaviour",
        "kind": "open-question",
        "severity": "significant",
        "status": "open",
        "description": "Emulating one GPU's rounding on another requires reverse-engineering tensor-core arithmetic and modelling proprietary kernel choices. Hawkeye covers a subset of NVIDIA architectures and states that attention and other higher-level operations need further reverse engineering [[S-1010]]. For the bit-exact emulator, a proprietary Hopper kernel family is an open edge case [[S-0020]].",
        "sources": [
          "S-1010",
          "S-0020"
        ],
        "response": null
      }
    ],
    "blockers": [
      {
        "text": "Batch-invariant kernels cost throughput: in Thinking Machines' Qwen3-8B test, an improved deterministic build took 42 s against 26 s for vLLM's default, and SGLang reports an average 34.35% slowdown on its FlashInfer and FlashAttention 3 backends.",
        "theme": "performance-compatibility",
        "blocked_by": null,
        "sources": [
          "S-1009",
          "S-1012"
        ]
      },
      {
        "text": "Coverage is incomplete: the bit-exact emulator targets dense blocks on NVIDIA GPUs and excludes mixture-of-experts inference and training; SGLang's deterministic mode supported dense models and one- or two-GPU tensor parallelism as of September 2025.",
        "theme": "performance-compatibility",
        "blocked_by": null,
        "sources": [
          "S-0020",
          "S-1012"
        ]
      },
      {
        "text": "A reproducible inference stack for verification has not been built; Amodo rates it 'not started'.",
        "theme": "performance-compatibility",
        "blocked_by": null,
        "sources": [
          "S-1008"
        ]
      },
      {
        "text": "Exact replay requires the prover to disclose weights, software versions, parallelism and batch sizes to whoever recomputes.",
        "theme": "privacy-leakage",
        "blocked_by": null,
        "sources": [
          "S-0020",
          "S-0018"
        ]
      }
    ],
    "challenge_themes": [
      "performance-compatibility",
      "protocol-soundness",
      "privacy-leakage",
      "adversarial-validation"
    ],
    "organizations": [],
    "people": [],
    "sources": [
      {
        "source": "S-0020",
        "supports": "covert-adversary threat model; deterministic but non-invariant engines; required metadata; software emulator and its results; limitations; comparison with statistical schemes; best-paper award and public code (arXiv comments)",
        "locator": "abstract; §1; results; limitations section; arXiv comments"
      },
      {
        "source": "S-0015",
        "supports": "logging of inferences and random sampling for verification as components separate from recomputation",
        "locator": "§5"
      },
      {
        "source": "S-0018",
        "supports": "replay metadata in a low-trust verification system",
        "locator": "§5.2.2"
      },
      {
        "source": "S-0016",
        "supports": "sources of benign nondeterminism; fixed-hardware determinism vs heterogeneous deployments; >98% token agreement",
        "locator": "§2; §3; §7"
      },
      {
        "source": "S-0017",
        "supports": "floating-point non-associativity; fuzzy comparison in recomputation schemes",
        "locator": "determinism discussion"
      },
      {
        "source": "S-0067",
        "supports": "reproducibility required for packet correctness checks",
        "locator": "Concrete inference-only retrofitting proposal"
      },
      {
        "source": "S-1008",
        "supports": "status of reproducible inference stack",
        "locator": "status items"
      },
      {
        "source": "S-1009",
        "supports": "batch invariance as main cause; kernels made invariant; Qwen3-235B experiment; timings",
        "locator": "whole post"
      },
      {
        "source": "S-1010",
        "supports": "exact CPU reproduction of tensor-core matrix multiplication; scope limits",
        "locator": "abstract; §8; §9"
      },
      {
        "source": "S-1011",
        "supports": "provider-reported end-to-end batch-invariant and deterministic kernels",
        "locator": "§3.3"
      },
      {
        "source": "S-1012",
        "supports": "SGLang deterministic mode, overhead and limitations",
        "locator": "whole post"
      },
      {
        "source": "S-1013",
        "supports": "vLLM batch-invariance flag, hardware requirement, beta status",
        "locator": "whole page"
      },
      {
        "source": "S-1014",
        "supports": "scheduling-based determinism alternative",
        "locator": "abstract"
      }
    ],
    "concepts": [
      "K-0008",
      "K-0009",
      "K-0016"
    ],
    "complements": [
      "M-0001",
      "M-0003",
      "M-0004"
    ],
    "alternatives": [],
    "type": "mechanism",
    "implementations": [
      {
        "id": "I-0012",
        "title": "Low-trust AI compute verification system overview",
        "url": "https://trustbutveri.fyi/implementations/low-trust-compute-verification-system-overview/"
      }
    ],
    "url": "https://trustbutveri.fyi/mechanisms/deterministic-inference/",
    "source_file": "content/mechanisms/deterministic-inference.md",
    "flags_all": [
      "ai-drafted"
    ],
    "body_markdown": "## How it works\n\nFloating-point arithmetic is not associative, so the same sum computed in a different order can round differently [[S-0017]] [[S-1009]]. In LLM serving, the order changes with [[S-0016]]:\n\n- batch size and kernel strategy, which depend on server load;\n- GPU type, CUDA version and kernel implementations;\n- in mixture-of-experts models, routing that depends on other tokens in the batch.\n\nThinking Machines Lab argues that the main reason inference endpoints are nondeterministic is that load, and so batch size, varies while kernels are not batch-invariant [[S-1009]]. The bit-exact work separates true nondeterminism, caused by atomic functions, from non-invariance: deterministic computation that follows different reduction trees [[S-0020]].\n\nThere are two routes to exact results:\n\n- **Invariant kernels.** Kernels fix the reduction order for each output element whatever the batch size [[S-1009]]. vLLM offers a batch-invariant mode, currently in beta [[S-1013]], and SGLang offers a deterministic inference mode [[S-1012]]. DeepSeek reports end-to-end bitwise batch-invariant and deterministic kernels, built with the goal of bitwise alignment among its pre-training, post-training and inference pipelines [[S-1011]]. LLM-42 enforces determinism through scheduling, replaying candidate tokens under a fixed reduction schedule instead of rewriting kernels [[S-1014]].\n- **Record and replay.** Stock engines already give deterministic outputs that a verifier can reproduce bit for bit, if the verifier knows the key factors and no atomic functions are called [[S-0020]]. The factors are the hardware model, exact weights, parallelism layout, software versions and the batch size of each forward pass [[S-0018]] [[S-0020]]. Software emulation removes the need for identical hardware [[S-0020]], and Hawkeye re-executes GPU matrix multiplications on a CPU without precision loss [[S-1010]].\n\nFor verification, exactness turns a recomputation check ([[M-0001]]) into a pass/fail test [[S-0020]]. [[M-0004|Zero-knowledge proofs of inference]] need determinism as a precondition [[S-0020]], and packet-based schemes ([[M-0003]]) need workloads to be reproducible [[S-0067]].\n\n## What it establishes\nUnder exact replay, the accumulated rounding errors become an auditable signature of the software and hardware used [[S-0020]]. The bit-exact work names three attacks that exploit the tolerance of approximate checks: steganography, unreported changes to inference software, and covert computation in unreported batch elements [[S-0020]]. It argues that statistical schemes can bound the covert bandwidth these leave, but cannot close it [[S-0020]].\n\nDeterminism does not capture traffic or choose samples; those come from recording and sampling mechanisms such as [[M-0013]] and [[M-0001]] [[S-0015]] [[S-0067]]. Batch-invariant kernels give identical results only while the model, inference implementation and device stay fixed [[S-0016]]. For varied inference stacks and mixed GPU types, DiFR's authors expect statistical verification to remain necessary [[S-0016]].\n\n## Threat model\n- **Adversary.** The bit-exact work targets covert adversaries, who comply with monitoring only when the chance of detection is high [[S-0020]].\n- **Full disclosure.** Exact replay assumes the verifier learns every factor that affects the numerics [[S-0020]]. A reference architecture for low-trust verification lists the same replay metadata [[S-0018]]. Recording the batch size is described as negligible overhead for the prover [[S-0020]].\n- **No atomic functions.** Backends must avoid atomic functions [[S-0020]].\n- **Correct hardware model.** Cross-hardware emulation assumes the hardware's rounding, subnormal handling and accumulation order have been characterized correctly [[S-1010]] [[S-0020]].\n\n## Evidence\n- **Nondeterminism measured.** For 1,000 temperature-0 completions of one prompt on Qwen3-235B-A22B, Thinking Machines reports 80 unique outputs with default kernels [[S-1009]]. With batch-invariant kernels, all 1,000 were identical [[S-1009]].\n- **Bit-exact emulation.** On Qwen3 4B blocks, the emulator reports zero BF16 differences for feed-forward blocks on A100, L40, L40S and H100 GPUs [[S-0020]]. It reports zero differences out of 71 million elements for FlashAttention-2 at 4,000 tokens [[S-0020]]. The paper received a best-paper award at the ICML 2026 TAIGR workshop, and its code is public [[S-0020]].\n- **Matrix multiplication on CPU.** Hawkeye, peer-reviewed at MLSys 2026, reports 100% success replicating 4096 × 4096 matrix multiplications on Ampere, Hopper and Lovelace GPUs [[S-1010]].\n- **Engines.** vLLM documents its batch-invariant mode [[S-1013]]. SGLang reports an average slowdown of 34.35% for its deterministic mode on FlashInfer and FlashAttention 3 backends [[S-1012]].\n- **Without determinism.** In DiFR's tests with synchronized seeds, over 98% of tokens already match exactly between provider and verifier [[S-0016]].\n\n## Limitations\n- **Throughput.** In Thinking Machines' test on Qwen3-8B, vLLM's default took 26 s, the unoptimized deterministic build 55 s, and the build with an improved attention kernel 42 s [[S-1009]].\n- **Coverage.** The emulator does not yet cover mixture-of-experts inference, non-NVIDIA GPUs, the proprietary nvjet kernel family on Hopper, or training [[S-0020]]. Hawkeye covers matrix multiplication only; attention and convolutions need further reverse engineering [[S-1010]]. As of its September 2025 post, SGLang supported deterministic inference only for dense models, and only with tensor parallelism across one or two GPUs [[S-1012]].\n- **Residual nondeterminism.** Some integer de-quantization kernels use atomic additions and remain truly nondeterministic [[S-0020]].\n- **Disclosure.** Replay requires exact weights and configuration details [[S-0020]].\n- **Maturity.** Amodo rates a reproducible inference stack for verification as not started, and red-teaming of recomputation schemes as not started [[S-1008]].",
    "body_text": "How it works Floating-point arithmetic is not associative, so the same sum computed in a different order can round differently [S-0017] [S-1009]. In LLM serving, the order changes with [S-0016]: - batch size and kernel strategy, which depend on server load; - GPU type, CUDA version and kernel implementations; - in mixture-of-experts models, routing that depends on other tokens in the batch. Thinking Machines Lab argues that the main reason inference endpoints are nondeterministic is that load, and so batch size, varies while kernels are not batch-invariant [S-1009]. The bit-exact work separates true nondeterminism, caused by atomic functions, from non-invariance: deterministic computation that follows different reduction trees [S-0020]. There are two routes to exact results: - Invariant kernels. Kernels fix the reduction order for each output element whatever the batch size [S-1009]. vLLM offers a batch-invariant mode, currently in beta [S-1013], and SGLang offers a deterministic inference mode [S-1012]. DeepSeek reports end-to-end bitwise batch-invariant and deterministic kernels, built with the goal of bitwise alignment among its pre-training, post-training and inference pipelines [S-1011]. LLM-42 enforces determinism through scheduling, replaying candidate tokens under a fixed reduction schedule instead of rewriting kernels [S-1014]. - Record and replay. Stock engines already give deterministic outputs that a verifier can reproduce bit for bit, if the verifier knows the key factors and no atomic functions are called [S-0020]. The factors are the hardware model, exact weights, parallelism layout, software versions and the batch size of each forward pass [S-0018] [S-0020]. Software emulation removes the need for identical hardware [S-0020], and Hawkeye re-executes GPU matrix multiplications on a CPU without precision loss [S-1010]. For verification, exactness turns a recomputation check (Sampled inference recomputation) into a pass/fail test [S-0020]. Zero-knowledge proofs of inference need determinism as a precondition [S-0020], and packet-based schemes (Reproducible computation packets) need workloads to be reproducible [S-0067]. What it establishes Under exact replay, the accumulated rounding errors become an auditable signature of the software and hardware used [S-0020]. The bit-exact work names three attacks that exploit the tolerance of approximate checks: steganography, unreported changes to inference software, and covert computation in unreported batch elements [S-0020]. It argues that statistical schemes can bound the covert bandwidth these leave, but cannot close it [S-0020]. Determinism does not capture traffic or choose samples; those come from recording and sampling mechanisms such as Network taps and certifiers and Sampled inference recomputation [S-0015] [S-0067]. Batch-invariant kernels give identical results only while the model, inference implementation and device stay fixed [S-0016]. For varied inference stacks and mixed GPU types, DiFR's authors expect statistical verification to remain necessary [S-0016]. Threat model - Adversary. The bit-exact work targets covert adversaries, who comply with monitoring only when the chance of detection is high [S-0020]. - Full disclosure. Exact replay assumes the verifier learns every factor that affects the numerics [S-0020]. A reference architecture for low-trust verification lists the same replay metadata [S-0018]. Recording the batch size is described as negligible overhead for the prover [S-0020]. - No atomic functions. Backends must avoid atomic functions [S-0020]. - Correct hardware model. Cross-hardware emulation assumes the hardware's rounding, subnormal handling and accumulation order have been characterized correctly [S-1010] [S-0020]. Evidence - Nondeterminism measured. For 1,000 temperature-0 completions of one prompt on Qwen3-235B-A22B, Thinking Machines reports 80 unique outputs with default kernels [S-1009]. With batch-invariant kernels, all 1,000 were identical [S-1009]. - Bit-exact emulation. On Qwen3 4B blocks, the emulator reports zero BF16 differences for feed-forward blocks on A100, L40, L40S and H100 GPUs [S-0020]. It reports zero differences out of 71 million elements for FlashAttention-2 at 4,000 tokens [S-0020]. The paper received a best-paper award at the ICML 2026 TAIGR workshop, and its code is public [S-0020]. - Matrix multiplication on CPU. Hawkeye, peer-reviewed at MLSys 2026, reports 100% success replicating 4096 × 4096 matrix multiplications on Ampere, Hopper and Lovelace GPUs [S-1010]. - Engines. vLLM documents its batch-invariant mode [S-1013]. SGLang reports an average slowdown of 34.35% for its deterministic mode on FlashInfer and FlashAttention 3 backends [S-1012]. - Without determinism. In DiFR's tests with synchronized seeds, over 98% of tokens already match exactly between provider and verifier [S-0016]. Limitations - Throughput. In Thinking Machines' test on Qwen3-8B, vLLM's default took 26 s, the unoptimized deterministic build 55 s, and the build with an improved attention kernel 42 s [S-1009]. - Coverage. The emulator does not yet cover mixture-of-experts inference, non-NVIDIA GPUs, the proprietary nvjet kernel family on Hopper, or training [S-0020]. Hawkeye covers matrix multiplication only; attention and convolutions need further reverse engineering [S-1010]. As of its September 2025 post, SGLang supported deterministic inference only for dense models, and only with tensor parallelism across one or two GPUs [S-1012]. - Residual nondeterminism. Some integer de-quantization kernels use atomic additions and remain truly nondeterministic [S-0020]. - Disclosure. Replay requires exact weights and configuration details [S-0020]. - Maturity. Amodo rates a reproducible inference stack for verification as not started, and red-teaming of recomputation schemes as not started [S-1008].",
    "referenced_by": [
      {
        "id": "M-0024",
        "title": "Bounding unexplained information in outputs",
        "url": "https://trustbutveri.fyi/mechanisms/bounding-unexplained-information/"
      },
      {
        "id": "M-0012",
        "title": "Model identity attestation",
        "url": "https://trustbutveri.fyi/mechanisms/model-identity-attestation/"
      },
      {
        "id": "M-0013",
        "title": "Network taps and certifiers",
        "url": "https://trustbutveri.fyi/mechanisms/network-taps-and-certifiers/"
      },
      {
        "id": "M-0003",
        "title": "Reproducible computation packets",
        "url": "https://trustbutveri.fyi/mechanisms/reproducible-computation-packets/"
      },
      {
        "id": "M-0001",
        "title": "Sampled inference recomputation",
        "url": "https://trustbutveri.fyi/mechanisms/sampled-inference-recomputation/"
      },
      {
        "id": "I-0012",
        "title": "Low-trust AI compute verification system overview",
        "url": "https://trustbutveri.fyi/implementations/low-trust-compute-verification-system-overview/"
      },
      {
        "id": "I-0008",
        "title": "SASH confidential network logger",
        "url": "https://trustbutveri.fyi/implementations/sash-confidential-network-logger/"
      },
      {
        "id": "C-0005",
        "title": "The declared model is the one being served",
        "url": "https://trustbutveri.fyi/claims/declared-model-is-served/"
      },
      {
        "id": "C-0009",
        "title": "Model weights or data have not left the facility",
        "url": "https://trustbutveri.fyi/claims/weights-have-not-left/"
      },
      {
        "id": "K-0008",
        "title": "Numerical nondeterminism",
        "url": "https://trustbutveri.fyi/concepts/numerical-nondeterminism/"
      }
    ]
  }
}