Deterministic and bit-exact inference

How it works

Floating-point arithmetic is not associative, so the same sum computed in a different order can round differently ^{5 8}. In LLM serving, the order changes with ⁴:

• batch size and kernel strategy, which depend on server load;
• GPU type, CUDA version and kernel implementations;
• in mixture-of-experts models, routing that depends on other tokens in the batch.

Thinking Machines Lab argues that the main reason inference endpoints are nondeterministic is that load, and so batch size, varies while kernels are not batch-invariant ⁸. The bit-exact work separates true nondeterminism, caused by atomic functions, from non-invariance: deterministic computation that follows different reduction trees ¹.

There are two routes to exact results:

• Invariant kernels. Kernels fix the reduction order for each output element whatever the batch size ⁸. vLLM offers a batch-invariant mode, currently in beta ¹², and SGLang offers a deterministic inference mode ¹¹. DeepSeek reports end-to-end bitwise batch-invariant and deterministic kernels, built with the goal of bitwise alignment among its pre-training, post-training and inference pipelines ¹⁰. LLM-42 enforces determinism through scheduling, replaying candidate tokens under a fixed reduction schedule instead of rewriting kernels ¹³.
• Record and replay. Stock engines already give deterministic outputs that a verifier can reproduce bit for bit, if the verifier knows the key factors and no atomic functions are called ¹. The factors are the hardware model, exact weights, parallelism layout, software versions and the batch size of each forward pass ^{3 1}. Software emulation removes the need for identical hardware ¹, and Hawkeye re-executes GPU matrix multiplications on a CPU without precision loss ⁹.

For verification, exactness turns a recomputation check (Sampled inference recomputation) into a pass/fail test ¹. Zero-knowledge proofs of inference need determinism as a precondition ¹, and packet-based schemes (Reproducible computation packets) need workloads to be reproducible ⁶.

What it establishes

Under exact replay, the accumulated rounding errors become an auditable signature of the software and hardware used ¹. The bit-exact work names three attacks that exploit the tolerance of approximate checks: steganography, unreported changes to inference software, and covert computation in unreported batch elements ¹. It argues that statistical schemes can bound the covert bandwidth these leave, but cannot close it ¹.

Determinism does not capture traffic or choose samples; those come from recording and sampling mechanisms such as Network taps and certifiers and Sampled inference recomputation ^{2 6}. Batch-invariant kernels give identical results only while the model, inference implementation and device stay fixed ⁴. For varied inference stacks and mixed GPU types, DiFR's authors expect statistical verification to remain necessary ⁴.

Threat model

• Adversary. The bit-exact work targets covert adversaries, who comply with monitoring only when the chance of detection is high ¹.
• Full disclosure. Exact replay assumes the verifier learns every factor that affects the numerics ¹. A reference architecture for low-trust verification lists the same replay metadata ³. Recording the batch size is described as negligible overhead for the prover ¹.
• No atomic functions. Backends must avoid atomic functions ¹.
• Correct hardware model. Cross-hardware emulation assumes the hardware's rounding, subnormal handling and accumulation order have been characterized correctly ^{9 1}.

Evidence

• Nondeterminism measured. For 1,000 temperature-0 completions of one prompt on Qwen3-235B-A22B, Thinking Machines reports 80 unique outputs with default kernels ⁸. With batch-invariant kernels, all 1,000 were identical ⁸.
• Bit-exact emulation. On Qwen3 4B blocks, the emulator reports zero BF16 differences for feed-forward blocks on A100, L40, L40S and H100 GPUs ¹. It reports zero differences out of 71 million elements for FlashAttention-2 at 4,000 tokens ¹. The paper received a best-paper award at the ICML 2026 TAIGR workshop, and its code is public ¹.
• Matrix multiplication on CPU. Hawkeye, peer-reviewed at MLSys 2026, reports 100% success replicating 4096 × 4096 matrix multiplications on Ampere, Hopper and Lovelace GPUs ⁹.
• Engines. vLLM documents its batch-invariant mode ¹². SGLang reports an average slowdown of 34.35% for its deterministic mode on FlashInfer and FlashAttention 3 backends ¹¹.
• Without determinism. In DiFR's tests with synchronized seeds, over 98% of tokens already match exactly between provider and verifier ⁴.

Limitations

• Throughput. In Thinking Machines' test on Qwen3-8B, vLLM's default took 26 s, the unoptimized deterministic build 55 s, and the build with an improved attention kernel 42 s ⁸.
• Coverage. The emulator does not yet cover mixture-of-experts inference, non-NVIDIA GPUs, the proprietary nvjet kernel family on Hopper, or training ¹. Hawkeye covers matrix multiplication only; attention and convolutions need further reverse engineering ⁹. As of its September 2025 post, SGLang supported deterministic inference only for dense models, and only with tensor parallelism across one or two GPUs ¹¹.
• Residual nondeterminism. Some integer de-quantization kernels use atomic additions and remain truly nondeterministic ¹.
• Disclosure. Replay requires exact weights and configuration details ¹.
• Maturity. Amodo rates a reproducible inference stack for verification as not started, and red-teaming of recomputation schemes as not started ⁷.