The data center network was designed for a specific traffic pattern, and that pattern shaped every design decision that followed, from the switching fabric to the cabling standards to the physical layout of the floor. Applications sitting on servers communicated with users outside the facility. Traffic flowed north to south, from the internet into the rack and back out again. That pattern shaped how networks were built, how switching fabrics were configured, and how bandwidth was allocated. AI neoclouds have, consequently, broken it entirely.
When CoreWeave, Lambda Labs, and Nebius build infrastructure for large AI training runs, the dominant traffic pattern is, consequently, not north-south. It is east-west, between GPU nodes within the cluster. A training job distributing gradients across tens of thousands of GPUs generates internal traffic that dwarfs anything passing through the facility’s external interfaces. The network, consequently, has to be rebuilt from the assumptions up.
Why Traditional Networks Break Under AI Workloads
A conventional enterprise or colocation data center network is built around a spine-and-leaf architecture optimised for north-south traffic. The switching fabric is sized to move data between servers and the outside world efficiently. East-west traffic between servers within the same facility was historically modest, generated mainly by distributed databases and virtualisation workloads.
AI training clusters reverse that ratio. InfiniBand or high-speed Ethernet fabrics between GPU nodes carry hundreds of gigabits per second of gradient traffic during a training run. The latency requirements are, notably, strict. A millisecond of additional round-trip time between GPU nodes can degrade training efficiency measurably. The network is, therefore, not a supporting element of the AI workload. It is, in other words, a performance-critical component as important as compute or memory. Traditional data center networks were not designed to this specification. Neoclouds, which emerged specifically to serve AI training workloads, built their networks to it from the start. As we have covered in our analysis of why bare metal GPU access is becoming the neocloud’s strongest selling point, the network architecture underneath the GPU is one of the primary reasons neoclouds perform differently from hyperscaler cloud GPU offerings for training workloads.
The Inference Layer Is Adding a Different Problem
The network complexity does not stop at training. Inference workloads, which are now growing faster than training by volume, have, however, different network requirements. Inference requests are typically independent of each other. They do not require the all-to-all GPU communication that makes training networks so demanding. What inference requires instead is low-latency connectivity between the inference server and the user generating the request. That is, in fact, a different network problem entirely.
Neoclouds that built their network for training are, consequently, discovering that inference at scale requires retrofitting a different architecture onto the same physical infrastructure. The tension between these two network patterns is, in turn, creating what operators call a dual-network problem. One fabric optimises for training, another for inference, both needing to coexist in the same physical space. As we have covered in our analysis of inference at scale being the neocloud’s next battlefield after training, the shift from training-first to inference-first workload mix is reshaping neocloud infrastructure decisions at every layer.
The Optical Shift That Is Making Network Constraints Worse
The network architecture challenge is compounded by the shift from copper to optical interconnects that is accelerating across AI infrastructure. Nvidia’s commitment to fiber-optic interconnects for rack-scale systems reflects a broader industry move driven by GPU cluster bandwidth and latency requirements. Optical interconnects carry more bandwidth over longer distances with lower latency than copper. That makes them, specifically, better suited for the east-west traffic patterns of AI training. They require, however, different switching infrastructure, different cabling standards, and different failure modes than copper-based networks. Neoclouds that are building new facilities from scratch can, consequently, design for optical from the start. Traditional operators retrofitting existing facilities face a more complex transition.
What This Means for Traditional Data Center Operators
The traffic pattern disruption has specific implications for traditional colocation and hyperscale operators who are now competing for AI workloads. Their existing network infrastructure, optimised for conventional workloads, is not well suited to the all-to-all GPU communication requirements of large training clusters. Retrofitting for AI training network requirements is, consequently, more expensive and complex than simply installing GPU servers in existing racks.
The operators who recognised this early built dedicated AI zones with InfiniBand fabrics and the physical co-location density that training clusters require. Those who treated AI GPUs as drop-in replacements for conventional compute are, notably, discovering that the network is the constraint. As we have covered in our analysis of neoclouds redefining competition beyond hyperscalers, the neocloud’s competitive advantage has never been solely about having GPUs. It has been about having the complete infrastructure stack, including network architecture, that AI workloads actually require. The network inside the AI data center is, ultimately, not a plumbing problem. It is a first-principles design problem that determines whether the compute investment underneath it can actually deliver its rated performance. The neoclouds that got there first are defining the answer. Traditional operators are, consequently, catching up on a timeline that is shorter than anyone anticipated two years ago.
The Performance Gap Is Already Visible
The operators who understand that the network is a performance-critical design constraint, and not commodity plumbing, are the ones building AI infrastructure that actually delivers on its rated specifications. Those who treat it as plumbing are building facilities that will underperform regardless of how many GPUs are installed. The performance gap between a network-first AI facility and a network-afterthought one is, consequently, not theoretical. It shows up in training time, in inference latency, and in the cost per useful compute cycle. Those are, ultimately, the metrics that matter to the enterprise buying AI infrastructure. Everything else is, frankly, just marketing.
