Deterministic and bit-exact inference

How it works

Floating-point arithmetic is not associative, so the same sum computed in a different order can round differently ^{5 8}. In LLM serving, the order changes with ⁴:

• batch size and kernel strategy, which depend on server load;
• GPU type, CUDA version and kernel implementations;
• in mixture-of-experts models, routing that depends on other tokens in the batch.

Thinking Machines Lab argues that the main reason inference endpoints are nondeterministic is that load, and so batch size, varies while kernels are not batch-invariant ⁸. The bit-exact work separates true nondeterminism, caused by atomic functions, from non-invariance: deterministic computation that follows different reduction trees ¹.

There are three routes to exact results:

• Batch-invariant kernels fix the reduction order for each output element whatever the batch size ⁸. vLLM offers a batch-invariant mode, currently in beta ¹², and SGLang offers a deterministic inference mode ¹¹. DeepSeek reports end-to-end bitwise batch-invariant and deterministic kernels, built with the goal of bitwise alignment among its pre-training, post-training and inference pipelines ¹⁰. LLM-42 enforces determinism through scheduling, replaying candidate tokens under a fixed reduction schedule, and mostly reuses existing kernels unchanged ¹³.

• Record and replay uses stock engines, which already give deterministic outputs that a verifier can reproduce bit for bit, if the verifier knows the key factors and no atomic functions are called ¹. The factors are the hardware model, exact weights, parallelism layout, software versions and the batch size of each forward pass ^{1 3}. Software emulation removes the need for identical hardware ¹, and Hawkeye re-executes GPU matrix multiplications on a CPU without precision loss ⁹.

• Reproducible operators fix the order of floating-point operations across hardware. Gensyn's RepOps uses this route so honest providers and a referee can reproduce the same result ¹⁶.

For verification, exactness turns a recomputation check into a pass/fail test ¹. Zero-knowledge proofs of inference need determinism as a precondition ¹, and packet-based schemes need workloads to be reproducible ⁶.

What it establishes

Under exact replay, the accumulated rounding errors become an auditable signature of the software and hardware used ¹. The bit-exact work names three attacks that exploit the tolerance of approximate checks: steganography, unreported changes to inference software, and covert computation in unreported batch elements on engines that are not batch-invariant ¹. It argues that statistical schemes can bound the covert bandwidth these leave, but cannot close it ¹.

Determinism does not capture traffic or choose samples; those come from recording and sampling mechanisms such as Network taps and certifiers and Sampled inference recomputation ^{2 6}. Batch-invariant kernels make batch size undetectable from outputs alone, so outputs no longer reveal unreported batch elements ¹. They give identical results only while the model, inference implementation and device stay fixed ⁴.

For varied inference stacks and mixed GPU types, DiFR's authors expect statistical verification to remain necessary ⁴. TAO, peer-reviewed at EuroSys 2026, is a tolerance-based alternative for heterogeneous hardware ²⁴. It accepts each operator's output if it falls within bounds derived from IEEE-754 worst-case error and empirical profiles, and settles disputes with a Merkle-anchored dispute game ²⁴. Its authors deployed its contract layer on an Ethereum testnet ²⁴.

Threat model

• Adversary. The bit-exact work targets covert adversaries, who comply with monitoring only when the chance of detection is high ¹.
• Full disclosure. Exact replay assumes the verifier learns every factor that affects the numerics ¹. A reference architecture for low-trust verification lists the same replay metadata ³. Recording the batch size is described as negligible overhead for the prover ¹.
• No atomic functions. Backends must avoid atomic functions ¹.
• Correct hardware model. Cross-hardware emulation assumes the hardware's rounding, subnormal handling and accumulation order have been characterized correctly ^{1 9}.