{ "schema_version": "1.0.0", "rubric_version": "1.0", "license": "CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)", "record": { "id": "M-0014", "slug": "bandwidth-limits-and-compartmentalization", "title": "Bandwidth limits and compartmentalization", "aliases": [ "Traffic shaping", "Isolated inference units", "Interconnect limits" ], "status": "draft", "last_reviewed": "2026-09-23", "review_interval_days": 90, "steward": null, "provenance": { "drafted_by": "ai", "reviewed_by": [] }, "risk_flags": [], "flags": [], "one_liner": "Capping or removing the network links between groups of accelerators, so that serving models still works but large training runs become impractically slow.", "summary": "Training frontier models needs very fast links between many accelerators; serving a trained model needs far less. This mechanism caps or removes the links between groups of accelerators (\"pods\"), so that inference keeps working but large training runs become slow and costly. The AI 2040 plan would remove much back-end networking to create isolated inference units. Lucid Computing's design caps each 72-GPU pod at 1 Gbps in each direction. Lucid estimates that covert frontier training would then be at least about 350 times less efficient, provided an auditor randomizes how pods connect to routers; with operator-chosen routing, it can fall to about 25 times. As of September 2026 no cap that a verifier can check has been built or red-teamed. The main obstacles are showing that all traffic crosses the capped links and that the enforcing devices can be trusted. The largest known weakness is progress in low-communication training.", "technical": "- **Pod and cap.** Lucid Computing's Traffic Shaping design models a pod as one NVIDIA GB200 NVL72 rack (72 GPUs). A shaper at each pod's north-south uplink enforces a recommended cap of β = 1 Gbps (125 MB/s) per direction per pod, averaged over 60 seconds, about 14 Mbps per GPU [[S-1301]].\n- **Adversary.** The reference served model is a \"Fable-sized model (estimated 6T parameter MoE)\"; the adversary's target is about 10x larger (60T parameters). Strategies analysed: centralized training adapted to the cap, DiLoCo-family low-communication data parallelism (including streaming and quantized variants), and SWARM-family decentralized pipeline parallelism. The analysis grants the adversary perfect overlap of communication and computation, zero pipeline bubbles, zero straggler and fault overhead, free coordination compute outside the pods, aggressive low-precision recipes and the best published communication-reduction techniques [[S-1301]].\n- **Results.** The headline is a lower bound on the inefficiency multiplier of about 350x, with central estimates of 1,000x–10,000x depending on training architecture. For SWARM-style pipelines the floor falls to about 140x only if two further unproven techniques both succeed (8x lossless activation compression at frontier scale and an unusually narrow 60T-parameter architecture). The appendix summary claims at least 100x, with roughly 1.4–3.5x of margin, after stacking every charitable and speculative assumption [[S-1301]].\n- **Conditions.** The auditor randomizes the assignment of pods to routers and re-randomizes it periodically, and per-pod local storage is declared, capped and inspected. If the operator could instead assign whole cells of 100 or more pods to one pipeline stage, the bound would collapse to about 90–220x uncompressed and as low as about 25x with compression [[S-1301]].\n- **Amodo's node-level limiter.** NVIDIA BlueField-3 DPUs replace the NICs on a 400G Ethernet link, reaching 390 Gbps unencrypted and 193 Gbps encrypted between two DPUs [[S-1313]].", "category": "isolation-architecture", "secondary_categories": [ "off-chip-devices" ], "verifies": [ { "claim": "C-0008", "role": "primary", "note": "Caps or removes communication between declared groups of accelerators (S-0067, S-1301)." }, { "claim": "C-0004", "role": "supporting", "note": "Intended to leave inference workable while making large training impractical (S-0067, S-1301)." }, { "claim": "C-0007", "role": "supporting", "note": "Bounds the size of model that can be trained efficiently across pods (S-1301)." }, { "claim": "C-0009", "role": "supporting", "note": "A cap on outgoing bandwidth bounds how much weight data can leave a facility in a given time (S-1508)." } ], "threat_model": "adversarial", "adversarial_evaluation": "analysis", "hardware_requirement": "retrofit-device", "prover_cooperation": "required", "confidentiality": "preserving", "depends_on": [ { "target": "M-0013", "note": "Monitored links are needed to show that all traffic leaving a group crosses the capped boundary." }, { "target": "M-0017", "note": "Shaping and monitoring devices must resist tampering and bypass." } ], "readiness": { "assessment": true, "level": "R1", "rubric_version": "1.0", "rationale": "R1: detailed designs are public, but no cap that a verifier can check has been built.\n\n- **R1** met: the AI 2040 plan proposes isolated inference units created by removing back-end networking, on the grounds that high-bandwidth links are mostly needed only for training [[S-0067]]. Lucid Computing gives a detailed design with a cap, a pod definition, adversary strategies, favourable assumptions and residual risks [[S-1301]]. The MIRI overview describes perimeters of monitored links sized to pods for inference [[S-0018]].\n- **R2** not met: Lucid states that the design is \"still at the design stage and not yet implemented or red-teamed\" [[S-1301]]. Amodo has demonstrated DPU-enforced rate limiting on 400G links on real hardware [[S-1313]]. That work targets weight security under a cooperating operator, not a cap that a verifier checks against the operator, and Amodo has not yet fully analysed resilience to a compromised DPU [[S-1313]]. The mechanism's implementations, [[I-0010]] and [[I-0011]], are proposed architectures at R1.\n\nConfidence is medium: the design evidence is clear, but it is uncertain whether node-level rate-limiting demonstrations should count toward this use.", "evidence": [ "S-0067", "S-1301", "S-0018", "S-1313" ], "next_level_gaps": [ "A public implementation of a pod-level cap that a verifier can check, tested on data-centre hardware (Lucid reports a proof of concept in development).", "Measured, not only modelled, training inefficiency under the cap, including low-communication methods.", "Red-teaming of cap bypass (for example through parallel scale-up switches) and of trust in the shaping devices." ], "confidence": "medium", "assessed_by": [ "ai-draft" ], "assessed_on": "2026-09-23", "status": "current", "dispute": null }, "flaws": [ { "assessment": true, "title": "Low-communication training reduces the bandwidth training needs", "kind": "theoretical-argument", "severity": "significant", "status": "open", "description": "DiLoCo matched fully synchronous training on 8 workers while communicating 500 times less. Rahman writes that this family of methods theoretically allows large-scale training with less than 100 Mbps. Lucid includes these methods in its bounds, but notes that extreme activation compression, architectures with unusually small inter-layer widths, or modular paradigms could erode the margin.", "sources": [ "S-1314", "S-0060", "S-1301" ], "response": null }, { "assessment": true, "title": "Operator control of pod routing collapses the bound", "kind": "theoretical-argument", "severity": "significant", "status": "open", "description": "Lucid's analysis finds that if the operator can freely assign pods to routers, it could dedicate a whole cell of 100 or more pods to one pipeline stage. The bound then falls to about 90–220x uncompressed and as low as about 25x with compression. The proposed mitigation, auditor-controlled random assignment that is periodically re-randomized, has not been implemented.", "sources": [ "S-1301" ], "response": null }, { "assessment": true, "title": "Undeclared local storage raises per-pod capacity", "kind": "theoretical-argument", "severity": "significant", "status": "open", "description": "More memory or storage per pod helps an adversary. Lucid requires per-pod storage to be declared, capped and physically inspected.", "sources": [ "S-1301" ], "response": null }, { "assessment": true, "title": "Training within one pod is not covered", "kind": "open-question", "severity": "significant", "status": "open", "description": "Lucid's bounds concern pre-training models larger than the pods are sized for. Training models that fit in one pod, fine-tuning and reinforcement-learning post-training within one pod are outside the modelled threat.", "sources": [ "S-1301" ], "response": null }, { "assessment": true, "title": "Parallel scale-up switches are hard enforcement points", "kind": "theoretical-argument", "severity": "significant", "status": "open", "description": "In GB200 topologies, GPUs reach GPUs in other nodes through NVSwitches without a NIC on the path. Amodo notes that limits are hard to enforce there because many switches work in parallel, so compromising one or two would bypass the limit.", "sources": [ "S-1313" ], "response": null } ], "blockers": [ { "text": "No cap that a verifier can check has been implemented or red-teamed.", "theme": "adversarial-validation", "blocked_by": null, "sources": [ "S-1301" ] }, { "text": "The verifier must know that all traffic leaving a pod crosses the capped, monitored links.", "theme": "coverage-hidden-compute", "blocked_by": "M-0013", "sources": [ "S-0018" ] }, { "text": "Shaping devices and routing assignments must be trusted by both parties; Amodo has not yet fully analysed resilience to a compromised DPU.", "theme": "hardware-trust", "blocked_by": "M-0017", "sources": [ "S-1301", "S-1313" ] }, { "text": "Advances in low-communication training could shrink the margin that the cap enforces.", "theme": "capacity-bounds", "blocked_by": null, "sources": [ "S-1314", "S-0060", "S-1301" ] } ], "challenge_themes": [ "capacity-bounds", "coverage-hidden-compute", "hardware-trust", "adversarial-validation", "performance-compatibility" ], "organizations": [ "O-0101", "O-0180", "O-0201", "O-0202" ], "people": [], "sources": [ { "source": "S-0067", "supports": "isolated inference units by removing back-end networking; rationale; retrofit illustration", "locator": "inference-only retrofitting proposal" }, { "source": "S-1301", "supports": "Traffic Shaping design, cap, pod model, adversary strategies and assumptions, results and their conditions, residual risks, status", "locator": "summary; main text; appendices A.7–A.9" }, { "source": "S-1508", "supports": "egress limits cap how much can be stolen", "locator": "§5.1" }, { "source": "S-0018", "supports": "perimeter of monitored links; perimeter size for inference and training", "locator": "§5.1.1" }, { "source": "S-1313", "supports": "DPU-based node bandwidth boundary, throughput, NVSwitch limitation, compromised-DPU caveat, security purpose", "locator": "whole note" }, { "source": "S-1314", "supports": "500x less communication on 8 workers", "locator": "abstract" }, { "source": "S-0060", "supports": "DiLoCo-family bandwidth needs; bandwidth caps on evaders judged infeasible", "locator": "§1; appendix G.1" } ], "concepts": [ "K-0017", "K-0021", "K-0025", "K-0016", "K-0018" ], "complements": [ "M-0005", "M-0001", "M-0003", "M-0015", "M-0016" ], "alternatives": [], "type": "mechanism", "implementations": [ { "id": "I-0011", "title": "AI 2040 inference-only verification stack", "url": "https://trustbutveri.fyi/implementations/ai-2040-inference-only-verification-plan/" }, { "id": "I-0010", "title": "RAND secure inference data center (SIDC) design", "url": "https://trustbutveri.fyi/implementations/rand-secure-inference-data-centers/" } ], "url": "https://trustbutveri.fyi/mechanisms/bandwidth-limits-and-compartmentalization/", "source_file": "content/mechanisms/bandwidth-limits-and-compartmentalization.md", "flags_all": [ "ai-drafted" ], "body_markdown": "## How it works\n\nTraining and serving use the network differently [[S-0067]] [[S-1301]]. The AI 2040 plan notes that AI servers \"have a lot of high bandwidth communication between them, but this is (mostly) only necessary for training\" [[S-0067]]. Lucid Computing gives the same rationale: inference sends little external traffic, while training moves gradients, model state and activations among many machines [[S-1301]].\n\nTwo families of design use this asymmetry [[S-0067]] [[S-1301]]:\n\n- **Removing links.** The AI 2040 plan would \"convert datacenters into groups of isolated inference-units\" by removing a significant amount of back-end networking [[S-0067]]. Its illustration retrofits 40% of US compute into inference units of four server racks [[S-0067]].\n- **Capping links.** Lucid's Traffic Shaping design installs a network shaper at the north-south uplink of each pod, modelled as one GB200 NVL72 rack [[S-1301]]. It recommends a cap of 1 Gbps per direction per pod, averaged over 60 seconds [[S-1301]]. The parties would configure and agree on the shaping devices, built on commodity hardware or open standards through a mediated co-design process [[S-1301]].\n\nThe MIRI system overview frames the same boundary as a perimeter: untrusted devices are grouped so that only monitored links communicate in and out [[S-0018]]. For inference, that perimeter can be as small as pods of a few hundred accelerators [[S-0018]]. Monitored links are covered in [[M-0013]].\n\nAmodo has built a related enforcement point for weight security [[S-1313]]. It replaced a node's NICs with NVIDIA BlueField-3 DPUs, reasoning that \"all the traffic in and out of a node goes through a NIC\", which creates \"a clean bandwidth boundary\" [[S-1313]].\n\nFor proofs of compartmentalization, a related approach to constraining training, see [[M-0005]].\n\n## What it establishes\nLucid aims to let a third party verify that frontier training either is not happening in a declared facility or would cost so much more that training a model ten times the current frontier becomes economically infeasible [[S-1301]]. It estimates a lower bound of about 350x on the inefficiency of covert frontier training, with central estimates of 1,000x to 10,000x, if the auditor controls pod routing and per-pod storage is capped [[S-1301]]. This addresses [[C-0008]], and indirectly [[C-0004]] and [[C-0007]].\n\nIt does not establish:\n\n- **What the pods compute.** A cap alone does not show this, so the AI 2040 plan pairs inference units with taps and recomputation ([[M-0001]], [[M-0003]]) [[S-0067]].\n- **Absence of training inside one pod.** Lucid places training that fits within one pod, fine-tuning and reinforcement-learning post-training within one pod outside its modelled threat [[S-1301]].\n- **Anything outside declared facilities.** Rahman judges it \"infeasible to prevent evaders from getting fast internet connections\" [[S-0060]].\n\n## Threat model\n- **Favourable adversary.** Lucid's analysis grants the adversary perfect overlap of communication and computation, zero pipeline bubbles and \"the best published communication-reduction techniques\", among other advantages [[S-1301]].\n- **Auditor-controlled routing.** The bound holds only if the auditor randomizes the assignment of pods to routers and periodically re-randomizes it [[S-1301]].\n- **Declared storage.** Per-pod storage must be declared, capped and inspected [[S-1301]].\n- **Complete perimeter.** The shaper sits at each pod's uplink [[S-1301]]. The verifier must also know that no other path leaves the pod, which the MIRI overview handles with a perimeter where only monitored links communicate in and out [[S-0018]]; see [[M-0013]] and [[M-0017]].\n\n## Evidence\n- **Designs.** The AI 2040 plan, Lucid's design brief and the MIRI overview are designs and analysis [[S-0067]] [[S-1301]] [[S-0018]]. Lucid describes its design as \"still at the design stage and not yet implemented or red-teamed\" [[S-1301]]. It reports that its engineers are building a proof of concept for red-teaming at a partner cluster [[S-1301]].\n- **Node-level limiter.** Amodo reports 390 Gbps unencrypted and 193 Gbps encrypted throughput between two DPUs on a 400G link [[S-1313]]. That demonstration targets weight security with a cooperating operator, and Amodo has \"not yet fully analysed resilience to a single compromised DPU\" [[S-1313]].\n\n## Limitations\n- **Low-communication training.** DiLoCo matched fully synchronous training on 8 workers while communicating 500 times less [[S-1314]]. Rahman writes that this family theoretically allows large-scale training at under 100 Mbps [[S-0060]]. Lucid's bounds include DiLoCo- and SWARM-family methods, but it notes that extreme activation compression or modular paradigms could erode the margin [[S-1301]].\n- **Routing control.** If the operator can group pods freely behind routers, the bound falls to about 90–220x uncompressed and as low as about 25x with compression [[S-1301]].\n- **Hidden storage.** Undeclared per-pod storage weakens the bound [[S-1301]].\n- **Scale-up bypass.** In GB200 topologies, GPUs reach other nodes through NVSwitches without a NIC on the path, and compromising one or two parallel switches would bypass a limit [[S-1313]].\n- **Durability.** Lucid positions the design as \"a short-to-medium-term deterrent and Phase-1 milestone\" [[S-1301]].", "body_text": "How it works Training and serving use the network differently [S-0067] [S-1301]. The AI 2040 plan notes that AI servers \"have a lot of high bandwidth communication between them, but this is (mostly) only necessary for training\" [S-0067]. Lucid Computing gives the same rationale: inference sends little external traffic, while training moves gradients, model state and activations among many machines [S-1301]. Two families of design use this asymmetry [S-0067] [S-1301]: - Removing links. The AI 2040 plan would \"convert datacenters into groups of isolated inference-units\" by removing a significant amount of back-end networking [S-0067]. Its illustration retrofits 40% of US compute into inference units of four server racks [S-0067]. - Capping links. Lucid's Traffic Shaping design installs a network shaper at the north-south uplink of each pod, modelled as one GB200 NVL72 rack [S-1301]. It recommends a cap of 1 Gbps per direction per pod, averaged over 60 seconds [S-1301]. The parties would configure and agree on the shaping devices, built on commodity hardware or open standards through a mediated co-design process [S-1301]. The MIRI system overview frames the same boundary as a perimeter: untrusted devices are grouped so that only monitored links communicate in and out [S-0018]. For inference, that perimeter can be as small as pods of a few hundred accelerators [S-0018]. Monitored links are covered in Network taps and certifiers. Amodo has built a related enforcement point for weight security [S-1313]. It replaced a node's NICs with NVIDIA BlueField-3 DPUs, reasoning that \"all the traffic in and out of a node goes through a NIC\", which creates \"a clean bandwidth boundary\" [S-1313]. For proofs of compartmentalization, a related approach to constraining training, see Zero-knowledge proofs of training constraints. What it establishes Lucid aims to let a third party verify that frontier training either is not happening in a declared facility or would cost so much more that training a model ten times the current frontier becomes economically infeasible [S-1301]. It estimates a lower bound of about 350x on the inefficiency of covert frontier training, with central estimates of 1,000x to 10,000x, if the auditor controls pod routing and per-pod storage is capped [S-1301]. This addresses Communication between compute groups is bounded, and indirectly This compute runs inference, not training and A training run stayed within declared limits. It does not establish: - What the pods compute. A cap alone does not show this, so the AI 2040 plan pairs inference units with taps and recomputation (Sampled inference recomputation, Reproducible computation packets) [S-0067]. - Absence of training inside one pod. Lucid places training that fits within one pod, fine-tuning and reinforcement-learning post-training within one pod outside its modelled threat [S-1301]. - Anything outside declared facilities. Rahman judges it \"infeasible to prevent evaders from getting fast internet connections\" [S-0060]. Threat model - Favourable adversary. Lucid's analysis grants the adversary perfect overlap of communication and computation, zero pipeline bubbles and \"the best published communication-reduction techniques\", among other advantages [S-1301]. - Auditor-controlled routing. The bound holds only if the auditor randomizes the assignment of pods to routers and periodically re-randomizes it [S-1301]. - Declared storage. Per-pod storage must be declared, capped and inspected [S-1301]. - Complete perimeter. The shaper sits at each pod's uplink [S-1301]. The verifier must also know that no other path leaves the pod, which the MIRI overview handles with a perimeter where only monitored links communicate in and out [S-0018]; see Network taps and certifiers and Tamper evidence for verifier devices. Evidence - Designs. The AI 2040 plan, Lucid's design brief and the MIRI overview are designs and analysis [S-0067] [S-1301] [S-0018]. Lucid describes its design as \"still at the design stage and not yet implemented or red-teamed\" [S-1301]. It reports that its engineers are building a proof of concept for red-teaming at a partner cluster [S-1301]. - Node-level limiter. Amodo reports 390 Gbps unencrypted and 193 Gbps encrypted throughput between two DPUs on a 400G link [S-1313]. That demonstration targets weight security with a cooperating operator, and Amodo has \"not yet fully analysed resilience to a single compromised DPU\" [S-1313]. Limitations - Low-communication training. DiLoCo matched fully synchronous training on 8 workers while communicating 500 times less [S-1314]. Rahman writes that this family theoretically allows large-scale training at under 100 Mbps [S-0060]. Lucid's bounds include DiLoCo- and SWARM-family methods, but it notes that extreme activation compression or modular paradigms could erode the margin [S-1301]. - Routing control. If the operator can group pods freely behind routers, the bound falls to about 90–220x uncompressed and as low as about 25x with compression [S-1301]. - Hidden storage. Undeclared per-pod storage weakens the bound [S-1301]. - Scale-up bypass. In GB200 topologies, GPUs reach other nodes through NVSwitches without a NIC on the path, and compromising one or two parallel switches would bypass a limit [S-1313]. - Durability. Lucid positions the design as \"a short-to-medium-term deterrent and Phase-1 milestone\" [S-1301].", "referenced_by": [ { "id": "M-0024", "title": "Bounding unexplained information in outputs", "url": "https://trustbutveri.fyi/mechanisms/bounding-unexplained-information/" }, { "id": "M-0009", "title": "Hardware-enabled guarantees (flexHEG) and guarantee processors", "url": "https://trustbutveri.fyi/mechanisms/flexheg-guarantee-processors/" }, { "id": "M-0015", "title": "Memory wiping and proofs of secure erasure", "url": "https://trustbutveri.fyi/mechanisms/memory-wiping-and-secure-erasure/" }, { "id": "M-0013", "title": "Network taps and certifiers", "url": "https://trustbutveri.fyi/mechanisms/network-taps-and-certifiers/" }, { "id": "M-0003", "title": "Reproducible computation packets", "url": "https://trustbutveri.fyi/mechanisms/reproducible-computation-packets/" }, { "id": "M-0022", "title": "Side-channel suppression for isolated facilities", "url": "https://trustbutveri.fyi/mechanisms/side-channel-suppression/" }, { "id": "M-0017", "title": "Tamper evidence for verifier devices", "url": "https://trustbutveri.fyi/mechanisms/tamper-evidence-for-verifier-devices/" }, { "id": "M-0016", "title": "Timed challenge-response and memory-occupation challenges", "url": "https://trustbutveri.fyi/mechanisms/timed-challenge-response/" }, { "id": "I-0011", "title": "AI 2040 inference-only verification stack", "url": "https://trustbutveri.fyi/implementations/ai-2040-inference-only-verification-plan/" }, { "id": "I-0010", "title": "RAND secure inference data center (SIDC) design", "url": "https://trustbutveri.fyi/implementations/rand-secure-inference-data-centers/" }, { "id": "C-0008", "title": "Communication between compute groups is bounded", "url": "https://trustbutveri.fyi/claims/bandwidth-is-bounded/" }, { "id": "C-0004", "title": "This compute runs inference, not training", "url": "https://trustbutveri.fyi/claims/inference-not-training/" }, { "id": "K-0017", "title": "Compartmentalization", "url": "https://trustbutveri.fyi/concepts/compartmentalization/" }, { "id": "K-0025", "title": "Inference and training workloads", "url": "https://trustbutveri.fyi/concepts/inference-and-training-workloads/" }, { "id": "K-0021", "title": "Interconnect bandwidth", "url": "https://trustbutveri.fyi/concepts/interconnect-bandwidth/" }, { "id": "K-0003", "title": "Positive and negative claims", "url": "https://trustbutveri.fyi/concepts/positive-and-negative-claims/" }, { "id": "O-0201", "title": "AI Futures Project", "url": "https://trustbutveri.fyi/organizations/ai-futures-project/" }, { "id": "O-0101", "title": "Amodo Design", "url": "https://trustbutveri.fyi/organizations/amodo-design/" }, { "id": "O-0180", "title": "Lucid Computing", "url": "https://trustbutveri.fyi/organizations/lucid-computing/" }, { "id": "O-0202", "title": "Machine Intelligence Research Institute", "url": "https://trustbutveri.fyi/organizations/machine-intelligence-research-institute/" } ] } }