Implementation · Batch-invariant inference kernels (Thinking Machines)
Sources
On this page
- CH. He & Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism. Source recordSupports: main cause of nondeterminism; operations made batch-invariant; fixed split size for attention; vLLM integration; Qwen3-235B and Qwen3-8B results; on-policy RL motivation (provider-reported) · whole post
- BThinking Machines Lab (2025). thinking-machines-lab/batch_invariant_ops (GitHub repository). GitHub. Source recordSupports: MIT-licensed library, replaced PyTorch operations, deterministic vLLM example · README
- CThe SGLang Team (2025). Towards Deterministic Inference in SGLang and Reproducible RL Training. LMSYS Org blog. Source recordSupports: SGLang's integration of the kernels, its attention kernels, the 61.5% figure for the original post and SGLang's average slowdown · whole post
- BvLLM project (2026). Batch Invariance (vLLM documentation). vLLM documentation (GitHub, docs/features/batch_invariance.md). Source recordSupports: vLLM batch-invariant mode, beta status, supported hardware (NVIDIA compute capability 8.0+, Intel XPUs), tested model families · whole page
- CvLLM project contributors (2025). [Feature]: Batch Invariant Feature and Performance Optimization (vLLM issue #27433). GitHub (vllm-project/vllm issues). Source recordSupports: vLLM developers' statement that batch-invariance support is based on the Thinking Machines post; open work items
- BA. Karvonen et al. (2025). DiFR: Inference Verification Despite Nondeterminism. ICML 2026 Workshop on Technical AI Governance Research. Source recordSupports: identical results only with fixed model, implementation and device; statistical verification for heterogeneous stacks
- BA. Arun et al. (2025). Verde: Verification via Refereed Delegation for Machine Learning Programs. arXiv. Source recordSupports: RepOps, Gensyn's own reproducible operators (rationale comparison) · §3.2
- BD. Ribeiro Alves et al. (2026). EigenAI: Deterministic Inference, Verifiable Results. arXiv. Source recordSupports: EigenAI's own deterministic kernels (rationale comparison; provider-reported) · abstract; kernels section