{
  "schema_version": "1.0.0",
  "rubric_version": "1.1",
  "license": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)",
  "record": {
    "id": "I-0016",
    "slug": "batch-invariant-inference-kernels",
    "title": "Batch-invariant inference kernels (Thinking Machines)",
    "aliases": [
      "batch_invariant_ops",
      "VLLM_BATCH_INVARIANT",
      "SGLang deterministic inference"
    ],
    "status": "draft",
    "last_reviewed": "2026-09-24",
    "review_interval_days": 90,
    "steward": null,
    "provenance": {
      "drafted_by": "ai",
      "reviewed_by": []
    },
    "risk_flags": [],
    "flags": [
      "provider-reported"
    ],
    "one_liner": "Open-source kernels from Thinking Machines Lab that make LLM outputs independent of batch size, adopted in vLLM and SGLang to give reproducible inference.",
    "summary": "Batch-invariant kernels make a language model give the same output for the same input whatever else the server is processing. Thinking Machines Lab identified changing batch sizes as the main reason LLM endpoints are nondeterministic, and published kernels whose reduction order does not depend on the batch. With them, 1,000 temperature-zero completions from a 235-billion-parameter model were identical. SGLang built its deterministic mode on the kernels, and vLLM's batch-invariant mode, still in beta, is based on the same work. For verification, identical outputs let a verifier that runs the same model, engine and hardware require an exact match instead of a tolerance. The kernels were built for reproducibility, not verification, and no verification system is documented using them. They cost throughput: 61.5% in the original test and 34% on average in SGLang.",
    "category": "cryptographic-computational",
    "secondary_categories": [],
    "verifies": [
      {
        "claim": "C-0005",
        "role": "supporting",
        "note": "Makes exact-match recomputation of served outputs possible when the verifier runs the same model, engine and hardware (S-0016)."
      }
    ],
    "threat_model": "cooperative",
    "adversarial_evaluation": "none",
    "hardware_requirement": "none",
    "prover_cooperation": "required",
    "confidentiality": "revealing",
    "depends_on": [],
    "readiness": {
      "assessment": true,
      "level": "R2",
      "rubric_version": "1.1",
      "rationale": "R2. Public kernels made a 235-billion-parameter model's outputs identical across 1,000 runs and underpin the deterministic modes of two production engines, but no verification system is documented using them.\n\n- **R1** met: Thinking Machines sets out the design and the property it gives, outputs that do not depend on batch size [[S-1009]]. The condition for identical outputs, a fixed model, inference implementation and device, is stated in the verification literature [[S-0016]].\n- **R2** met: the code is public under an MIT licence, with a deterministic vLLM example [[S-1813]]. With the kernels, 1,000 temperature-zero completions from Qwen3-235B were identical, against 80 unique completions without them [[S-1009]].\n- **R3** not met for this use. The kernels have reached production engines: SGLang builds its deterministic mode on them [[S-1012]], and vLLM's developers state that its batch-invariant mode, labelled beta, is based on the same work [[S-1814]] [[S-1013]]. The kernels were built for reproducibility, such as on-policy reinforcement learning [[S-1009]]. As for [[M-0002]], no production-grade verification stack uses them, and no party is documented relying on them for a verification decision.\n- **R4** not met: no independent security evaluation has been published.\n\nConfidence is low, because whether an engine's deterministic mode counts as production-grade for verification is a judgment call.",
      "evidence": [
        "S-1009",
        "S-1813",
        "S-1012",
        "S-1013",
        "S-1814",
        "S-0016"
      ],
      "next_level_gaps": [
        "A production-grade verification stack that uses batch-invariant inference for exact-match checks, or a party other than the developers relying on such checks for a verification decision."
      ],
      "confidence": "low",
      "assessed_by": [
        "ai-draft"
      ],
      "assessed_on": "2026-09-24",
      "status": "current",
      "dispute": null
    },
    "flaws": [],
    "blockers": [
      {
        "text": "Batch invariance costs throughput: on Qwen3-8B the improved deterministic build took 42 s against 26 s for vLLM's default, and SGLang reports an average slowdown of 34.35% on its FlashInfer and FlashAttention 3 backends.",
        "theme": "performance-compatibility",
        "blocked_by": null,
        "sources": [
          "S-1009",
          "S-1012"
        ]
      },
      {
        "text": "Outputs are identical only while the model, inference implementation and device stay fixed, so provider and verifier must run the same stack.",
        "theme": "performance-compatibility",
        "blocked_by": null,
        "sources": [
          "S-0016"
        ]
      }
    ],
    "challenge_themes": [
      "performance-compatibility",
      "adversarial-validation",
      "privacy-leakage"
    ],
    "organizations": [],
    "people": [],
    "sources": [
      {
        "source": "S-1009",
        "supports": "main cause of nondeterminism; operations made batch-invariant; fixed split size for attention; vLLM integration; Qwen3-235B and Qwen3-8B results; on-policy RL motivation (provider-reported)",
        "locator": "whole post"
      },
      {
        "source": "S-1813",
        "supports": "MIT-licensed library, replaced PyTorch operations, deterministic vLLM example",
        "locator": "README"
      },
      {
        "source": "S-1012",
        "supports": "SGLang's integration of the kernels, its attention kernels, the 61.5% figure for the original post and SGLang's average slowdown",
        "locator": "whole post"
      },
      {
        "source": "S-1013",
        "supports": "vLLM batch-invariant mode, beta status, hardware requirement, tested model families",
        "locator": "whole page"
      },
      {
        "source": "S-1814",
        "supports": "vLLM developers' statement that batch-invariance support is based on the Thinking Machines post; open work items"
      },
      {
        "source": "S-0016",
        "supports": "identical results only with fixed model, implementation and device; statistical verification for heterogeneous stacks"
      }
    ],
    "concepts": [
      "K-0008",
      "K-0009"
    ],
    "kind": "open-source-project",
    "developer": [],
    "realises": [
      "M-0002"
    ],
    "homepage": "https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/",
    "repo": "https://github.com/thinking-machines-lab/batch_invariant_ops",
    "type": "implementation",
    "url": "https://trustbutveri.fyi/implementations/batch-invariant-inference-kernels/",
    "source_file": "content/implementations/batch-invariant-inference-kernels.md",
    "flags_all": [
      "provider-reported",
      "ai-drafted"
    ],
    "body_markdown": "## What it is\n\nThinking Machines Lab published batch-invariant kernels for LLM inference in September 2025 [[S-1009]]. Its post argues that \"the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies\" [[S-1009]]. The companion library, batch_invariant_ops, is MIT-licensed [[S-1813]].\n\nThe kernels are one route to [[M-0002|deterministic and bit-exact inference]]. SGLang built its deterministic mode on them [[S-1012]], and vLLM's developers state that its batch-invariant mode is based on the post [[S-1814]].\n\n## How it works\n\nA kernel is batch-invariant if it computes each output element with the same reduction order whatever the batch size [[S-1009]]. The post identifies three operations that involve reductions: RMSNorm, matrix multiplication and attention [[S-1009]]. For attention it fixes the size of each split along the key-value dimension, rather than the number of splits [[S-1009]].\n\nThe library replaces PyTorch's matrix multiplication (torch.mm and torch.addmm), log-softmax and mean through torch.Library, so existing models can use the kernels with small code changes [[S-1813]]. The post runs the result in vLLM through its FlexAttention backend [[S-1009]]. vLLM now exposes its own mode with the environment variable VLLM_BATCH_INVARIANT=1, on NVIDIA GPUs of compute capability 8.0 or higher [[S-1013]].\n\nFor verification, identical outputs hold when the model, inference implementation and device are fixed [[S-0016]]. A verifier that runs the same stack can then require an exact match, instead of the tolerance that [[M-0001|sampled recomputation]] schemes allow.\n\n## Evidence\n- For 1,000 temperature-zero completions of one prompt on Qwen3-235B, Thinking Machines reports 80 unique outputs with default kernels and one with batch-invariant kernels [[S-1009]].\n- The library's vLLM example gave 18 unique samples out of 1,000 completions without the upstream vLLM change, and one with it [[S-1813]].\n- SGLang integrated the mean, log-softmax and matrix-multiplication kernels and wrote batch-invariant attention kernels for several backends. It reports an average slowdown of 34.35% on its FlashInfer and FlashAttention 3 backends, against the 61.5% in the original post [[S-1012]].\n- vLLM documents its batch-invariant mode as beta and lists tested model families including DeepSeek V3, Qwen3, Llama 3.1 and GPT-OSS [[S-1013]].\n\n## Limitations\n- Outputs are identical only while the model, inference implementation and device stay fixed. For varied stacks and GPU types, DiFR's authors expect statistical verification to remain necessary [[S-0016]].\n- In Thinking Machines' Qwen3-8B test, vLLM's default took 26 s, the unoptimised deterministic build 55 s and the build with an improved attention kernel 42 s [[S-1009]].\n- vLLM's mode is in beta, and its tracking issue lists open work on performance, AMD hardware and speculative decoding [[S-1013]] [[S-1814]].\n- The post's motivation is reproducibility, including on-policy reinforcement learning, not verification against a cheating provider [[S-1009]].",
    "body_text": "What it is Thinking Machines Lab published batch-invariant kernels for LLM inference in September 2025 [S-1009]. Its post argues that \"the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies\" [S-1009]. The companion library, batchinvariantops, is MIT-licensed [S-1813]. The kernels are one route to deterministic and bit-exact inference. SGLang built its deterministic mode on them [S-1012], and vLLM's developers state that its batch-invariant mode is based on the post [S-1814]. How it works A kernel is batch-invariant if it computes each output element with the same reduction order whatever the batch size [S-1009]. The post identifies three operations that involve reductions: RMSNorm, matrix multiplication and attention [S-1009]. For attention it fixes the size of each split along the key-value dimension, rather than the number of splits [S-1009]. The library replaces PyTorch's matrix multiplication (torch.mm and torch.addmm), log-softmax and mean through torch.Library, so existing models can use the kernels with small code changes [S-1813]. The post runs the result in vLLM through its FlexAttention backend [S-1009]. vLLM now exposes its own mode with the environment variable VLLMBATCHINVARIANT=1, on NVIDIA GPUs of compute capability 8.0 or higher [S-1013]. For verification, identical outputs hold when the model, inference implementation and device are fixed [S-0016]. A verifier that runs the same stack can then require an exact match, instead of the tolerance that sampled recomputation schemes allow. Evidence - For 1,000 temperature-zero completions of one prompt on Qwen3-235B, Thinking Machines reports 80 unique outputs with default kernels and one with batch-invariant kernels [S-1009]. - The library's vLLM example gave 18 unique samples out of 1,000 completions without the upstream vLLM change, and one with it [S-1813]. - SGLang integrated the mean, log-softmax and matrix-multiplication kernels and wrote batch-invariant attention kernels for several backends. It reports an average slowdown of 34.35% on its FlashInfer and FlashAttention 3 backends, against the 61.5% in the original post [S-1012]. - vLLM documents its batch-invariant mode as beta and lists tested model families including DeepSeek V3, Qwen3, Llama 3.1 and GPT-OSS [S-1013]. Limitations - Outputs are identical only while the model, inference implementation and device stay fixed. For varied stacks and GPU types, DiFR's authors expect statistical verification to remain necessary [S-0016]. - In Thinking Machines' Qwen3-8B test, vLLM's default took 26 s, the unoptimised deterministic build 55 s and the build with an improved attention kernel 42 s [S-1009]. - vLLM's mode is in beta, and its tracking issue lists open work on performance, AMD hardware and speculative decoding [S-1013] [S-1814]. - The post's motivation is reproducibility, including on-policy reinforcement learning, not verification against a cheating provider [S-1009].",
    "referenced_by": [
      {
        "id": "M-0002",
        "title": "Deterministic and bit-exact inference",
        "url": "https://trustbutveri.fyi/mechanisms/deterministic-inference/"
      }
    ]
  }
}