Interconnect bandwidth

Interconnect bandwidth is the rate at which accelerators, servers or clusters can exchange data over the links between them; it is one of the measurable specifications of AI accelerators, alongside operations per second and memory capacity ¹.

Large-scale training links thousands of accelerators with high-bandwidth interconnect, while efficient inference can run on dozens to low hundreds of closely connected accelerators ². Between such pods, inference needs to pass only tokens, whereas training exchanges gradients or activations; Scher and Thiergart identify this gap as the target of bandwidth limits, as in bandwidth limits and compartmentalization ². Inside a data centre, front-end links carry token-level inputs and outputs, while the back-end fabric between accelerators carries tensors and collective operations at much higher bandwidth, is latency-sensitive, and is harder to tap ³. A US executive order defined reportable computing clusters partly by network connections faster than 100 Gbit/s ¹. The distinction between workloads could weaken if decentralized training over lower-quality compute and links becomes more viable ¹.