InfiniBand vs. RoCEv2: A Measured Comparison for Large-Scale AI Training Clusters

In multi-node GPU workloads such as large language model (LLM) pretraining, the real bottleneck is no longer the performance of a single GPU. When dozens of nodes have to exchange hundreds of gigabytes of gradients at every step, the inter-node network becomes the decisive factor in training speed. That is why the choice between InfiniBand and RoCEv2 (RDMA over Converged Ethernet) has to be settled early in the design of any GPU cluster.

Today those two technologies are the only realistic options for a large AI cluster. They come from different design philosophies and different ecosystems, and how that difference plays out under real AI training workloads varies enough between environments that field measurements matter more than spec sheets. Elice Cloud operates both an InfiniBand fabric and a RoCEv2-based Ethernet fabric, which let us compare collective communication performance quantitatively under identical NCCL benchmark conditions.

This post covers what we measured across both fabrics — up to 12 nodes / 96 GPUs for the common comparison range, and up to 30 nodes / 240 GPUs for the RoCEv2 scaling range — and what those differences mean in practice.

Key findings

• On bandwidth-dominated collectives (AllReduce, AllGather), RoCEv2 matched InfiniBand or led it by as much as 16%.
• On AlltoAll, InfiniBand held roughly a 4% edge — not enough to register in real workloads.
• Equipment required to build a 512-GPU fabric: RoCEv2 needed 96% fewer switches, 40% fewer transceivers, and 50% less fiber than InfiniBand.
• InfiniBand is effectively single-vendor (NVIDIA); RoCEv2 sits on a multi-vendor Ethernet ecosystem and the UEC standardization track.
  

Why the network matters again

In single-node training, GPU-to-GPU traffic stays inside NVLink and NVSwitch, so the external network never enters the picture. The moment you tie two or more nodes together, though, a single AllReduce makes dozens of round trips across the inter-node link, and the bandwidth and latency of that segment begin to govern total throughput.

The two mainstream configurations:

• InfiniBand is a dedicated interconnect with native RDMA support. Latency is low, congestion control is built into the protocol, and it has been proven in HPC environments for years. The trade-offs: it requires dedicated switches, cables, and NICs, it has to be operated as a separate fabric from existing Ethernet infrastructure, and NDR-generation switches and NICs depend in practice on a single supply chain — NVIDIA, since the Mellanox acquisition in 2020 — which carries vendor lock-in risk around availability, pricing, and roadmap.
• RoCEv2 runs RDMA over Ethernet. It is L3-routable and works with standard Ethernet switches, so it integrates with the rest of the data center network, and high-density Ethernet switches allow a simpler topology. Switch and NIC vendors are diversified across Broadcom, NVIDIA, Arista, Cisco, and Marvell, which gives procurement more room. In exchange, lossless transport requires up-front tuning of PFC (Priority Flow Control) and ECN (Explicit Congestion Notification).

Which one suits AI workloads better is not a question spec comparison can answer. It requires measuring collective communication performance directly.

Test environment

We ran the benchmark on two production Elice Cloud clusters using the same NCCL test scripts.

Both environments use eight ConnectX-7 NICs per node in a rail-optimized layout that assigns a dedicated NIC to each GPU, and we held link speed (400G), NCCL build, plugin, and tuning variables identical across the two.

The switch topologies differ, but that is not a deliberate handicap — each one reflects the minimum number of tiers its technology can use to house 512 GPUs.

• InfiniBand (two-tier leaf-spine): NDR-based Quantum-2 switches top out at 64 ports, so a single switch physically cannot cover 512 GPUs. A two-tier leaf-spine is therefore mandatory. This is an inherent constraint of IB, not a choice made to tilt the comparison.
• RoCEv2 (single-tier): Current Ethernet switches offer 576 ports × 800G in a single chassis, and with 400G breakout one switch can terminate up to 1,152 links at 400G. A 512-GPU fabric fits in a single tier with no additional layer.

The meaningful comparison range is 2 through 12 nodes. A single-node / 8-GPU configuration is served entirely by intra-node NVLink, with the inter-node network never involved, so we excluded the 1-node results from the analysis and interpret the difference between the two fabrics only from 2 nodes up.

What this looks like on the floor

The topology difference above is immediately visible in front of the rack. The photographs below show half of each 512-GPU fabric — the 256-GPU portion. The InfiniBand build needs a second, identical rack to complete 512 GPUs; the Ethernet build has only half of its switch links populated.

▲ Photo 1 — InfiniBand configuration

The two-tier leaf-spine concentrates a large number of switches and leaf-to-spine cables into a single rack. Cable density and transceiver count are high, and rack footprint and port management scale with them.

▲ Photo 2 — RoCEv2 (Ethernet) configuration

Because it completes in a single tier, the same scale fits in one chassis and there is no leaf-to-spine cabling at all. Cabling, transceiver count, wiring complexity, failure points, and operational overhead all drop.

The density difference in the photographs shows up in the parts list as well. The table below is the actual equipment used to build a non-blocking fabric at 512 GPUs (64 nodes × 8 NICs, 512 links at 400G).

Build details. The IB fabric is NDR Quantum-2 based. Each switch provides 32 OSFP cages (2×400G per port, 64 × 400G total), so we used single-port 400G OSFP transceivers on the server NIC side and dual-port 2×400G OSFP on the switch side. It is laid out as a rail-optimized non-blocking topology (16 leaf + 8 spine, 32 ports down / 32 ports up per rail), with two 400G cables connected independently to each dual-port module on the switch side so that every 400G link maps one-to-one to a physical cable. That produced 512 leaf-to-server strands and 512 leaf-to-spine strands.

The RoCEv2 fabric follows the same principle: in a single 576-port × 800G chassis, each 800G port is populated with a dual-port 2×400G OSFP, so 256 switch-side modules terminate 512 links at 400G directly into the server NICs.

Cutting switch count by 96% is not only a cost question. It means fewer failure points, simpler configuration management, and lower rack power and cooling load. With transceivers down 40% and physical cabling down 50%, the two builds differ structurally in long-term maintenance and wiring complexity as well as in up-front CAPEX.

Benchmark method

We measured network performance with NVIDIA's nccl-tests, covering all five collective communication patterns that appear in distributed AI training:

• AllReduce — the most frequent operation in distributed training, used to sum gradients
• AllGather — gathers each rank's parameters across all nodes (used by FSDP, ZeRO, and similar)
• AlltoAll — central to MoE expert routing and sequence parallelism
• Hypercube — a reference benchmark for topology-based exchange patterns
• SendRecv — point-to-point communication, used in pipeline parallelism

The metric is Bus Bandwidth (BusBW, GB/s), the standard measure that normalizes for algorithmic structure so you can compare how hard the network is actually working. We swept message sizes from 8 MB to 8 GB, doubling at each step, and report the 8 GB point as the representative value in the body of this post.

Results

Result 1 — AllReduce: the operation that matters most, and effectively a tie

AllReduce is the most frequent collective in LLM training. At an 8 GB message size, BusBW from 2 nodes up — where the inter-node network is actually engaged — came out as follows.

From 2 to 12 nodes the two fabrics stayed within 0.7–2.8% of each other, with RoCEv2 consistently ahead by a small margin at every point.

In other words, on a bandwidth-bound workload like AllReduce the two networks perform essentially identically. Given that AllReduce is the core operation of LLM pretraining, the real-world gap between them is not large.

Result 2 — AllGather: RoCEv2 ahead by roughly 15%

AllGather, which appears constantly in FSDP and ZeRO-3 training, produced a result we did not expect.

RoCEv2 recorded 14–17% higher BusBW at every multi-node point. Since the NIC (ConnectX-7) and link speed (400G) are identical, the difference does not come from link bandwidth. The likely cause is the difference in topology tiers (IB two-tier leaf-spine vs. RoCEv2 single-tier) and the congestion behavior each topology produces. Ring-based AllGather repeats its transfer stages in proportion to rank count, so variance in single-hop latency accumulates and shows up in performance — and in this environment the single-tier configuration appears to have been the better fit for that condition.

Given that AllGather is a basic building block of FSDP and ZeRO training, this suggests a RoCEv2 fabric can hold a meaningful advantage over InfiniBand in large-scale parameter-sharded training.

Result 3 — AlltoAll: a narrow InfiniBand win

AlltoAll, used in Mixture of Experts (MoE) routing and sequence parallelism, went the other way.

InfiniBand led by roughly 1–5% from 2 through 12 nodes. In AlltoAll every pair of nodes tries to communicate simultaneously, so in-network congestion control feeds directly into performance, and InfiniBand's credit-based flow control appears to behave a little more predictably under that condition.

The gap itself is around 4%. Even accounting for AlltoAll being the core communication pattern of MoE models, a difference that size is unlikely to register meaningfully in real training throughput.

Result 4 — Hypercube and SendRecv: no meaningful difference

For Hypercube and SendRecv, BusBW differences converged within ±5%.

Hypercube (8 GB)

† hypercube_perf does not produce results when the rank count is not a power of two (96 ranks, for example), so the 12-node point is excluded.

SendRecv (8 GB)

Hypercube varied by only −0.2 to +1.0% from 2 to 8 nodes, and SendRecv gave RoCEv2 an edge of about +4.5–4.7% at 8 and 12 nodes. In this territory, either choice produces roughly the same experience.

Result 5 — Behavior by message size: InfiniBand for small, RoCEv2 for large

Everything above uses the 8 GB representative value. Real workloads, though, distribute their communication across the whole 8 MB–8 GB range, and which fabric leads changes with message size. Taking AllReduce as the example, here is BusBW by size at 12 nodes / 96 GPUs, the largest point in the common comparison range.

The RoCEv2 drop at 32 MB is unusually large relative to its neighbors. We suspect NCCL algorithm selection, a buffer boundary, or RoCE congestion behavior reacting differently at that specific size, but we did not isolate the cause in this benchmark. Setting that one point aside, the broader pattern — InfiniBand ahead across the latency-dominated range — holds either way.

Three regimes emerge:

• 8 MB – 32 MB (latency-dominated): InfiniBand leads by 15–41%, with the caveat that 32 MB behaves as an outlier. For small messages, protocol round-trip latency and scheduling overhead dominate total time rather than transfer time, so InfiniBand's structural low-latency advantage comes through directly. The more GPUs involved, the more that latency accumulates, and the wider the gap gets.
• 64 MB – 512 MB (transition): The two fabrics oscillate, with one or the other leading by as much as 18% depending on the point. This is where message size begins to offset latency and the workload crosses into the bandwidth-dominated regime; fine-grained factors like routing and congestion behavior swing the result heavily.
• 1 GB and above (bandwidth-dominated): Both fabrics saturate in the 300–370 GB/s range, with RoCEv2 stably ahead by 2–8%. RoCEv2's narrow lead at 8 GB is the converged result of this saturation region.

This distinction supplies important context for reading the earlier results. In LLM pretraining, the dominant traffic — gradient AllReduce — scales with model size into the hundreds of megabytes to several gigabytes, and FSDP/ZeRO parameter AllGather mostly lands above 1 GB as well. The main communication range of real AI training therefore sits in the bandwidth-saturated region, where the two fabrics are equal or RoCEv2 is slightly ahead. Below a few tens of megabytes, in the latency-dominated region, InfiniBand's low-latency characteristics make a clear difference in real performance — and for HPC, scientific computing, or any workload heavy in small-message communication, that region cannot be waved away.

Result 6 — RoCEv2 scaling

We extended measurement on the RoCEv2 fabric past the 12-node common comparison range, up to 30 nodes / 240 GPUs. There is no InfiniBand counterpart at this scale, so what follows describes RoCEv2 scaling behavior on its own. BusBW at 8 GB message size:

The key points:

• AllReduce held nearly flat at roughly 367–370 GB/s from 2 to 16 nodes, and even at 24 and 30 nodes fell only to 323–344 GB/s — a decline of about 7–13%.
• AllGather followed the same pattern, holding around 372 GB/s to 16 nodes and converging to roughly 324 GB/s (−13%) at 30 nodes.
• AlltoAll flattened at 47–49 GB/s past 12 nodes, meaning bandwidth did not collapse further even as more nodes communicated simultaneously.
• SendRecv flattened completely at 17.67–17.70 GB/s past 12 nodes, with no further degradation as scale increased.

The RoCEv2 fabric, in other words, does not fall apart as it scales; its degradation converges within a predictable range.

Summary of results

Consolidating all five collective patterns across the 2–12 node common range:

On the bandwidth-dominated collectives that matter most (AllReduce, AllGather), RoCEv2 matches or beats InfiniBand. InfiniBand leads only on AlltoAll, and not by enough to be felt in practice. As Result 6 shows, the RoCEv2 fabric holds its BusBW degradation within a predictable range even scaled out to 30 nodes / 240 GPUs.

Conclusion

Taken together, InfiniBand and RoCEv2 emerge as two axes with different strengths for large-scale AI training. On the bandwidth-dominated collectives — AllReduce and AllGather — RoCEv2 matched or led InfiniBand. On AlltoAll and in the sub-100 MB message range, InfiniBand held a narrow edge. The AlltoAll gap of around 4% is small enough that it is hard to call it a difference users would feel.

With performance that close, the physical build cost and complexity is where RoCEv2 pulls clearly ahead. Building the same 512-GPU fabric took 96% fewer switches, 40% fewer transceivers, and 50% less physical cabling on the RoCEv2 side, which structurally reduces up-front CAPEX, failure points, operational load, and rack power. Integration with the rest of the Ethernet estate adds a further operational benefit.

The ecosystem and supply-chain difference is hard to ignore as well. With InfiniBand, NVIDIA is effectively the sole supplier since the Mellanox acquisition, which ties availability, pricing, and product roadmap to one vendor. RoCEv2 runs on an Ethernet ecosystem contested by Broadcom, NVIDIA, Arista, Cisco, and Marvell, and the Ultra Ethernet Consortium — with AMD, Intel, Meta, Microsoft, Broadcom, Cisco, and Arista participating — is driving AI-optimized Ethernet standardization, so the long-term technical direction is diversified too. That shift shows up directly in recent large-scale AI cluster builds: Meta (Llama training, RoCE), Microsoft Azure, Alibaba (HPN), ByteDance (MegaScale), and AWS (EFA) have all chosen Ethernet for new AI infrastructure, while InfiniBand persists in environments with a strong HPC tradition or where an existing InfiniBand fabric is being extended.

The most telling data point may be that NVIDIA itself now presents its Spectrum-X Ethernet switch family as the standard AI infrastructure fabric beginning with the next-generation Vera Rubin platform. That the company which effectively drove InfiniBand is moving the backbone of its next AI platform to Ethernet says something beyond product-line diversification — it marks the center of gravity in AI networking shifting toward Ethernet. How the gaps observed here change once Rubin-generation hardware and NVIDIA's Spectrum-X Ethernet fabric are widely deployed will be an interesting follow-up comparison.

To summarize:

• For workloads dominated by AllReduce and AllGather — LLM pretraining, fine-tuning, FSDP/ZeRO-based distributed training — RoCEv2 is the stronger choice across performance, operations, cost, and scalability.
• For MoE training with heavy AlltoAll traffic, or latency-sensitive HPC-style work, InfiniBand held a narrow edge, but the gap is small enough that it is unlikely to matter in practice.

Elice Cloud operates both InfiniBand and RoCEv2 clusters on ECI (Elice Cloud Infrastructure), its own cloud platform, so customers can choose the infrastructure that fits their workload. If you are planning a large-scale AI training project, you can validate your actual workload directly in an ECI environment.