What Is the Most Effective Way to Move Data Around an AI Data Center?
What Is the Most Effective Way to Move Data Around an AI Data Center?
The fastest AI data centers do not use one network for everything. They minimize unnecessary data movement and combine several specialized fabrics: ultra-fast scale-up links such as NVLink inside GPU domains, RDMA-based InfiniBand or AI-optimized Ethernet between racks, dedicated high-bandwidth storage networks, and optical fiber for longer connections. As AI clusters grow toward tens or hundreds of thousands of accelerators, silicon photonics and co-packaged optics are becoming increasingly important.
An AI data center has a strange problem.
The GPUs can perform extraordinary amounts of computation, but they are useful only if data reaches them quickly enough.
A large model may be divided across dozens, hundreds, or thousands of accelerators. During training or inference, those chips repeatedly exchange model parameters, activations, gradients, tokens, and intermediate results.
If the network is slow, enormously expensive GPUs simply sit there waiting.
That is why networking has become almost as important as the processors themselves.
The best solution is not to choose between Ethernet, InfiniBand, NVLink, copper, or fiber as though one technology must defeat all the others.
Modern AI infrastructure works best as a hierarchy, using each technology where its physical and economic characteristics make the most sense.
1. The First Rule Is to Avoid Moving Data Whenever Possible
AI infrastructure engineers spend enormous effort increasing network bandwidth, but one of the most effective optimizations happens before a packet enters the network.
Keep data close to the processors that need it.
This principle is known broadly as data locality.
If a training job constantly retrieves the same dataset from a distant storage cluster, network bandwidth is wasted repeatedly moving identical information across the facility.
Instead, modern systems use local NVMe storage, distributed caches, memory pools, and intelligent job placement so frequently used data remains close to the GPUs.
The same principle applies to distributed model training.
GPUs that communicate constantly should ideally be placed within the same high-bandwidth domain or in nearby racks rather than scattered across the data center.
Job schedulers increasingly consider network topology when allocating accelerators because physical distance can affect performance.
AI networking therefore begins with an odd rule: before making the network faster, ask whether the data needs to cross the network at all.
2. Inside the Rack, Ultra-Fast Scale-Up Interconnects Win
Within a rack-scale AI system, the objective is to make many accelerators behave almost like one enormous processor.
This is called scale-up networking.
NVIDIA's current GB200 and GB300 rack-scale systems use NVLink and NVSwitch for this job.
A GB200 NVL72 system connects 72 Blackwell GPUs within one NVLink domain. Fifth-generation NVLink provides up to 1.8 terabytes per second of bidirectional GPU-to-GPU bandwidth per GPU.
The entire NVLink switch system can provide roughly 130 terabytes per second of low-latency GPU communication across the rack.
These extremely short links are where electrical connectivity still makes enormous sense.
Copper, printed-circuit-board traces, and short electrical connections generally consume less power and introduce less complexity than converting every signal into light when the distance is only centimeters or a few meters.
The rule is therefore simple:
For extremely short distances, keep the path electrical and extremely fast.
As the distance grows, however, the physics becomes considerably less friendly.
3. Between Racks, RDMA Over InfiniBand or Ethernet Moves the Heavy Traffic
Once an AI workload grows beyond one rack, the networking problem changes.
Thousands of GPUs may participate in one training job, exchanging enormous bursts of data through collective operations such as all-reduce and all-to-all communication.
Traditional office-style Ethernet was never designed for this traffic pattern.
The industry therefore relies heavily on Remote Direct Memory Access, or RDMA.
RDMA allows a network adapter to place data directly into memory without forcing the operating system and CPU to copy the information repeatedly between buffers.
That reduces latency, CPU overhead, and unnecessary memory traffic.
InfiniBand has long been one of the preferred technologies for large AI training clusters because it was designed around low latency and high-performance computing.
NVIDIA's current Quantum-X800 InfiniBand platform provides up to 800 Gb/s connectivity per port and includes adaptive routing, congestion control, and hardware acceleration for collective operations.
Ethernet is becoming a much stronger competitor.
AI-oriented Ethernet systems such as NVIDIA Spectrum-X combine high-speed Ethernet with RDMA, adaptive routing, telemetry, and sophisticated congestion management.
The Ultra Ethernet Consortium is taking the same idea further through an open industry standard specifically designed for AI and high-performance computing.
Its transport architecture uses RDMA-style direct data placement, multipath networking, and new congestion-control mechanisms intended to keep thousands of simultaneous accelerator flows from blocking one another.
The practical choice is therefore no longer simply “InfiniBand good, Ethernet slow.”
Both can now serve high-performance AI clusters when the network is engineered specifically for accelerator traffic.
4. Optical Fiber Becomes More Important as Distance and Bandwidth Increase
Electrical signals become increasingly difficult to move efficiently as both distance and data rate increase.
At hundreds of gigabits per second, copper cables experience signal loss and require more sophisticated equalization and signal processing.
Those components consume electricity and produce additional heat.
Optical fiber changes the equation by transmitting information as photons rather than electrical signals.
Fiber provides enormous bandwidth over much greater distances and avoids many of the electrical losses associated with copper.
This is why optical transceivers are already common between switches and racks in large data centers.
But the industry is moving the optical conversion closer to the networking chip itself.
Co-packaged optics, or CPO, places optical engines alongside the switch silicon rather than relying on separate pluggable optical transceivers positioned at the front of the switch.
The objective is to reduce the distance high-speed electrical signals must travel before being converted to light.
NVIDIA's latest Spectrum-X Ethernet Photonics and Quantum-X InfiniBand Photonics systems use this approach.
The Spectrum-X photonics platform is designed for up to 409.6 terabits per second of total switch bandwidth.
This matters because network electricity consumption becomes substantial when tens of thousands of optical links are operating simultaneously.
As AI clusters move toward hundreds of thousands or even a million accelerators, saving power on every data connection becomes economically important.
5. Storage Traffic Should Not Fight GPU Traffic for the Same Network
One of the easiest ways to ruin an expensive AI cluster is to force every type of data through the same congested network.
GPU communication is highly sensitive to latency.
Storage traffic has different characteristics. Training systems may continuously stream enormous datasets, checkpoints, and model files between storage arrays and compute nodes.
Management traffic is different again. It needs reliability but relatively little bandwidth.
Modern reference architectures therefore separate these functions.
NVIDIA's current data-center reference architecture describes four distinct networking domains.
NVLink handles high-bandwidth scale-up communication within GPU racks.
A Cluster Interconnect Network provides GPU-to-GPU scale-out traffic between racks using InfiniBand or Ethernet.
A Tenant Access Network carries storage and general application traffic.
A separate Secure Management Network handles configuration and administrative functions.
This separation prevents a large storage transfer or management event from interfering with a time-sensitive GPU collective operation.
Direct-storage technologies and GPUDirect-style data paths can optimize the design further by reducing unnecessary copying through host CPUs.
The architecture sounds more complicated because it is more complicated. Thousands of synchronized accelerators apparently object to sharing the same network with someone's backup job.
Key Takeaways at a Glance
- Minimize movement first: Place workloads and frequently used data close to the GPUs that need them.
- Use scale-up links locally: NVLink and similar interconnects provide extremely high bandwidth and low latency within tightly coupled GPU domains.
- Use RDMA for scale-out: InfiniBand and AI-optimized Ethernet efficiently connect GPU racks across the data center.
- Use optics as distance grows: Fiber and silicon photonics become increasingly attractive for high-speed inter-rack and campus-scale connections.
- Separate traffic types: Compute, storage, management, and external traffic should use appropriately designed fabrics rather than competing for one network.
| Data Movement | Typical Best Approach |
|---|---|
| GPU-to-GPU inside a rack | NVLink / NVSwitch or another dedicated scale-up interconnect |
| GPU rack to GPU rack | RDMA over InfiniBand or AI-optimized Ethernet |
| Longer rack or campus connections | High-speed optical fiber |
| Storage to compute | Dedicated high-throughput storage fabric with direct data placement where possible |
| Management traffic | Separate Ethernet management network |
The Future of AI Computing Depends as Much on Communication as Computation
The enormous performance of modern GPUs creates an ironic problem.
The faster the processors become, the more expensive it becomes to leave them waiting for data.
That is why AI infrastructure design is increasingly moving beyond the simple question of how many accelerators a company owns.
Network topology, memory architecture, data placement, congestion management, optical technology, and storage design increasingly determine how much useful work those accelerators actually perform.
The most efficient architecture therefore uses a hierarchy.
Keep the most frequently communicating GPUs together.
Use ultra-fast electrical interconnects over the shortest distances.
Use RDMA-oriented fabrics when scaling across racks.
Move to optical fiber when electrical reach becomes inefficient.
Separate storage and management traffic from performance-critical compute traffic.
And avoid moving the same data repeatedly when local caching or smarter workload placement can eliminate the transfer entirely.
The next major transition is already underway.
800 Gb/s networking is becoming available for AI fabrics, Ethernet is being redesigned specifically for AI workloads, and the industry's roadmap is moving toward 1.6 Tb/s connections and more widespread silicon photonics.
Co-packaged optics will move the optical conversion progressively closer to the networking silicon, reducing the amount of high-speed electrical signaling required inside switches.
Eventually, optical links may move even closer to the processors themselves.
The AI data center of the future will therefore not merely contain faster computers.
It will increasingly resemble one enormous computer in which the distance between chips, memory, storage, racks, and buildings is treated as part of the computing architecture itself.
Sources
NVIDIA — DGX GB Rack Scale Systems Networking Guide.
NVIDIA — DGX SuperPOD Reference Architecture.
NVIDIA — GB200 NVL72 Architecture.
NVIDIA — Quantum-X800 InfiniBand Platform.
NVIDIA — Spectrum-X Ethernet Platform.
NVIDIA — Silicon Photonics Networking for Agentic AI.
Ultra Ethernet Consortium — Ultra Ethernet Specification 1.0.3, July 2026.
Nature npj Nanophotonics — Industry Insight: Photonics to Scale AI Data Centers, January 2026.
