Bandwidth limits and compartmentalization

How it works

Training and serving use the network differently ^{1 2}. The AI 2040 plan notes that AI servers "have a lot of high bandwidth communication between them, but this is (mostly) only necessary for training" ¹. Lucid Computing gives the same rationale: inference sends little external traffic, while training moves gradients, model state and activations among many machines ².

Two families of design use this asymmetry ^{1 2}:

• The AI 2040 plan would remove a significant amount of back-end networking to "convert datacenters into groups of isolated inference-units" ¹. Its illustration retrofits 40% of US compute into inference units of four server racks ¹.
• Lucid's Traffic Shaping design caps links with a network shaper at the north-south uplink of each pod, modelled as one GB200 NVL72 rack ². It recommends a cap of 1 Gbps per direction per pod, averaged over 60 seconds ². The parties would configure and agree on the shaping devices, built on commodity hardware or open standards through a mediated co-design process ².

The MIRI system overview frames the same boundary as a perimeter: untrusted devices are grouped so that only monitored links communicate in and out ⁴. For inference, that perimeter can be as small as pods of a few hundred accelerators ⁴. Monitored links are covered in Network taps and certifiers.

Amodo Design reports building a related enforcement point for weight security ⁷. It replaced a node's NICs with NVIDIA BlueField-3 DPUs, network devices with their own CPU and operating system, reasoning that "all the traffic in and out of a node goes through a NIC", which creates "a clean bandwidth boundary" ⁷.

Reuter and colleagues call a workload compartmentalized if it splits into a limited number of cheap computations, each with a small output and little input beyond information committed in advance, such as model weights ¹⁰. Inference splits this way into chat sessions. Frontier training does not, because its parameter updates are too large to pass between compartments ¹⁰. The authors specify two protocols, proofs of compartmentalization (PoComps), that check this property while allowing inference ¹⁰:

• Pod-PoComp treats each pod's computation between scheduled memory wipes as one compartment. Gateway logs account for traffic. The protocol limits compute and uncommitted input, and caps output within each compartment ¹⁰.
• Task-PoComp explains the gateway logs with small deterministic tasks and checks a random sample through zero-knowledge replay. Its input and output budgets apply to each task, so they can be sized to one inference request instead of a whole pod's traffic between wipes ¹⁰.

The paper gives protocol specifications, conditional soundness arguments and cost estimates. It reports a prototype of one component, deterministic inference, but no end-to-end PoComp implementation ¹⁰.

What it establishes

Lucid Computing aims to let a third party verify that frontier training either is not happening in a declared facility or would cost so much more that training a model ten times the current frontier becomes economically infeasible ². It estimates that covert frontier training would need about 350 times the GPU-hours of uncapped training at minimum, with central estimates of 1,000x to 10,000x, if the auditor controls pod routing and per-pod storage is capped ². For decentralized pipeline training, the floor falls to about 140x if two unproven techniques both succeed. Lucid calls its figures "engineering estimates, not proofs" ². The cap addresses bounded communication between compute groups, and indirectly inference-only use and declared training limits.

Reuter and colleagues target the absence of frontier training by limiting the computation, uncommitted input and output of each compartment ¹⁰. Their argument depends on how far parameter updates can be compressed or rebuilt from compact inputs ¹⁰. The paper notes that training very small models, such as recursive reasoners under 10 million parameters, and evolutionary algorithms over small genomes can fit within the limits ¹⁰.

It does not establish:

• A cap alone does not show what the pods compute, so the AI 2040 plan pairs inference units with taps and recomputation (Sampled inference recomputation, Whole-workload recomputation (reproducible packets)) ¹.
• Lucid places training that fits within one pod outside its modelled threat, along with fine-tuning and reinforcement-learning post-training within one pod ².
• The caps apply only inside declared facilities. Outside them, Rahman's study of distributed training judges it "infeasible to prevent evaders from getting fast internet connections" ⁹.

Threat model

• Favourable adversary. Lucid Computing's analysis grants the adversary perfect overlap of communication and computation, zero pipeline bubbles and "the best published communication-reduction techniques", among other advantages ².
• Auditor-controlled routing. The bound holds only if the auditor randomizes the assignment of pods to routers and periodically re-randomizes it ².
• Declared storage. Per-pod storage must be declared, capped and inspected ².
• Complete perimeter. The shaper sits at each pod's uplink ². The verifier must also know that no other path leaves the pod, which the MIRI system overview handles with a perimeter where only monitored links communicate in and out ⁴; see Network taps and certifiers and Tamper evidence for verifier devices.