Data center network topology has historically been designed around predictable traffic patterns. Enterprise applications generated relatively stable flows between clients and servers. Storage systems produced consistent read and write traffic that engineers could model with reasonable accuracy. Even the first generation of cloud workloads, despite their scale, followed patterns that allowed network architects to design around known demand profiles and provision capacity accordingly. The fat-tree and spine-leaf architectures that dominate modern data center networking emerged from these patterns, optimizing for east-west traffic distribution across compute clusters while maintaining low latency at manageable cost. Those architectures solved the right problems for the workloads they served. AI inference is a different problem entirely.
Inference workloads do not generate the sustained, predictable traffic patterns that conventional network architectures handle efficiently. A single inference request can trigger a cascade of internal data movements that bears no resemblance to the traffic profile of the request that initiated it. Retrieval-augmented generation systems, which combine real-time database lookups with model execution, create traffic bursts that move across multiple network segments simultaneously before converging on the inference engine. Multi-modal inference workloads that process text, image, and structured data inputs generate parallel traffic streams with different latency requirements that engineers must coordinate without introducing bottlenecks at any convergence point. The network topology that handles these patterns efficiently is not the one that most data centers currently operate.
The Traffic Model That Inference Breaks
Spine-leaf architectures distribute traffic across multiple equal-cost paths between leaf switches and spine switches, providing predictable bandwidth and low latency for east-west communication within a data center fabric. This architecture works efficiently when traffic flows remain relatively uniform and when latency requirements across different traffic types stay broadly similar. Inference workloads violate both assumptions simultaneously. The traffic that a high-throughput inference cluster generates is neither uniform nor latency-homogeneous. Some traffic flows require microsecond-level latency because they sit on the critical path of a live user request. Other flows involve bulk data movement for model weight updates or cache population that tolerates latency but demands high bandwidth.
Mixing these traffic types within a shared network fabric without explicit prioritization creates interference that degrades the latency-sensitive flows without meaningfully improving throughput on the bandwidth-intensive ones. The problem compounds as inference clusters scale. A single inference server handling modest request volumes generates manageable network traffic. A cluster of hundreds of inference servers handling tens of thousands of concurrent requests generates traffic patterns that stress the oversubscription ratios that conventional spine-leaf designs carry. Most spine-leaf deployments accept some degree of oversubscription at the spine layer, assuming that not all leaf-to-spine bandwidth will be simultaneously utilized. AI inference workloads at scale challenge this assumption because their traffic patterns correlate in ways that conventional workloads do not.
Latency Asymmetry and Its Network Implications
AI inference introduces a latency asymmetry between the request path and the response path that directly affects network buffer management and queuing strategy. Inference requests are typically small in data volume but require rapid delivery to the inference engine to minimize queuing delay. Inference responses can be substantially larger, particularly for generative tasks that produce long text outputs or image data, and their delivery requirements depend on whether the application streams output incrementally or delivers it as a complete response. These asymmetric traffic profiles require network designs that handle small, latency-sensitive inbound flows and larger, throughput-oriented outbound flows without queuing policies optimized for one direction degrading performance in the other.
Buffer management in conventional data center switches is typically configured for symmetric or near-symmetric traffic flows. Inference workloads expose the limitations of these configurations by generating sustained asymmetry that causes buffer pressure to concentrate on specific switch ports in predictable but difficult-to-mitigate ways. Network architects addressing this problem evaluate adaptive buffer allocation schemes that dynamically redistribute buffer capacity based on observed traffic asymmetry, but these approaches require switch hardware that supports dynamic buffer partitioning. This hardware is not universally available in the installed base of data center switches. The hardware refresh cycle that addressing this limitation requires adds to the total cost of adapting existing data center network infrastructure for inference workloads.
Disaggregated Inference and the Network Complexity It Creates
The most architecturally challenging dimension of inference network topology is the disaggregation of inference workloads across multiple specialized hardware components. Large language model inference at commercial scale rarely runs entirely on a single server. Attention computation, which dominates the computational profile of transformer-based models, has different hardware affinity than feed-forward computation, and the memory bandwidth requirements of serving large models create pressure to distribute model weights across multiple accelerators that high-bandwidth interconnects connect. When this disaggregation happens within a single server, it involves proprietary interconnect fabrics that operate below the network layer. When it spans multiple servers, it creates network traffic that differs qualitatively from conventional distributed compute traffic.
Disaggregated inference traffic carries strict latency requirements across inter-server communication because the partial computations happening on different servers must synchronize at each layer of the model. A disaggregated inference system running across four servers requires those servers to exchange intermediate activations at every transformer layer, generating a burst of synchronized traffic at microsecond intervals throughout the inference computation. Conventional Ethernet networks, even at high bandwidths, introduce jitter in this synchronized traffic that degrades inference throughput below what the aggregate compute capacity of the cluster would suggest. This mismatch between network capability and inference traffic requirements drives evaluation of alternative interconnect technologies for inference clusters that operators previously associated only with training workloads.
What Redesigned Inference Network Architecture Looks Like
Data center operators building purpose-built inference infrastructure move away from general-purpose spine-leaf designs toward topologies that explicitly separate traffic classes with different latency and bandwidth requirements. The most common approach creates dedicated network segments for latency-sensitive inference traffic and separate segments for bulk data movement, with explicit policies governing how traffic transitions between segments. This separation allows each segment to optimize for its specific traffic profile without the interference that mixed-traffic architectures produce. The capital cost of this separation is real, requiring additional switching infrastructure and more complex cabling plants than a unified fabric supports.
Software-defined networking plays an increasingly important role in inference network design by enabling traffic classification and path selection at the application layer rather than relying solely on hardware-level queue prioritization. Operators deploying inference at scale instrument their workloads to generate network telemetry that reflects inference-specific metrics, such as time-to-first-token and batch completion latency, rather than generic network performance metrics. This telemetry feeds control systems that dynamically adjust routing and queuing policies based on observed inference performance rather than predicted traffic patterns. The combination of purpose-built physical topology and software-defined traffic management produces network architectures that bear little resemblance to the general-purpose fabrics that hyperscale data centers operated in the training era, and the gap between inference-optimized and conventional network designs will widen as AI workload requirements continue to grow.
