Five Parallel Storage Platforms Benchmarked for AI Training — the Best Fit Depends on the Workload

In large-scale AI training, the GPU is not the only thing that decides how fast the job actually feels. As workloads diversify across checkpoint writes, training-data loading, and multimodal asset handling, the same storage behaves very differently depending on access pattern, block size, node scale, and transport protocol (TCP vs. RDMA). Picking "the fastest storage" off a single metric is getting harder by the year.

Elice Cloud connected five parallel storage platforms commonly used in AI/HPC infrastructure directly to ECI (Elice Cloud Infrastructure), our own cloud platform, and ran a PoC on all of them against the same criteria. The platforms: DDN AI400X3i, Weka Server, VAST Data Platform, IBM Storage Scale System 3200, and IBM Storage Scale System 6000. The goal was to confirm that each one integrates cleanly with ECI, and to understand its performance characteristics well enough to recommend the right configuration for a given customer workload.

PoC key findings

First, all five platforms integrated with ECI without issue and ran AI training workloads normally. That also means ECI is not tied to a single vendor — it works across the major parallel storage systems.

Second, each platform's strengths fall in clearly different places. The leaders in sequential read (checkpoint restore), sequential write (checkpoint save), and 4K random (DataLoader) were different every time, and we did not observe a platform that took both read and write. The conclusion is not that one storage system is absolutely superior, but that the right configuration changes with which workload dominates.

RDMA delivered throughput equal to or higher than TCP in most configurations. Where we saw it invert, the right reading is not a limit of the storage itself but a signal that the stack needs tuning.

One caveat matters. This PoC was validation work on small-scale configurations, so the observed numbers are not each platform's absolute performance — they are values bounded by the PoC equipment scale. Higher throughput is possible at production scale, so the numbers that follow should be read as per-workload performance characteristics rather than as "storage X delivers Y GiB/s."

Elice's answer: ECI, and AI PMDC

Building on the experience of integrating and operating multiple vendors' parallel storage on ECI — as in this PoC — Elice Group already offers parallel file systems on ECI as a production service. Create a file system at whatever capacity you need from the console, mount it straight onto a GPU virtual machine, and large-scale AI training can start immediately with no separate storage infrastructure to build.

If you have requirements public cloud alone cannot meet, the same ECI platform can be deployed on your own site and operated as a private or hybrid cloud. Where the data center infrastructure itself has to be built from scratch, AI PMDC (Portable Modular Data Center) covers that too — and in the process we design the storage configuration together with you, informed by the observations in this PoC, around your training dataset size, checkpoint patterns, and scaling plans.

Detailed technical report

From here on is the technical report: how we measured, and the results by workload. Each PoC was performed on the vendor's own reference configuration, and because these were small-scale validation builds, the throughput ceiling is bounded by the PoC equipment scale.

Parallel storage hardware used in the PoC

Reference configurations differ from vendor to vendor, so there is substantial variation in everything from scale to network generation.

Scale ranges widely — from a single 2U appliance (DDN, IBM SSS 3200) to a 12-node disaggregated cluster (VAST) — and the storage-side network spans generations from 200GbE to 400GbE. The observations below should therefore be read as performance characteristics on each reference hardware build, not as an absolute comparison.

To avoid the observation sections being misread as a ranking of specific products, we anonymize the vendors as A, B, C, D, and E. The mapping between A–E and the products above is not disclosed.

Why performance characteristics have to be verified by PoC

Storage enters AI training at four main points, each with different I/O characteristics.

• Training data loading: the DataLoader reading thousands to hundreds of millions of samples in parallel — random read / multi-file random read
• Checkpoint save: large writes of tens of gigabytes of model state — sequential write
• Checkpoint restore: reading that state back — large sequential read
• Log and metric writes: small, continuous writes during training — random write

It is difficult for one configuration to excel equally at all four. The "up to XX GB/s" figures quoted in vendor material are usually sequential-read numbers, which do not faithfully represent throughput as it is actually experienced. Choosing storage therefore means defining which workload dominates first, measuring performance in that specific regime, and only then deciding.

Test environment and FIO profiles

All five platforms were measured with the same FIO profiles, at 1 / 8 / 16-node scale, over both TCP and RDMA. Every PoC ran over Ethernet links, so RDMA in this post refers to RoCEv2 (RDMA over Converged Ethernet version 2), not InfiniBand RDMA. The client was ECI's G-NBTHS-1440 instance (8 × NVIDIA B200 GPU), so 1 / 8 / 16 nodes correspond to 8 / 64 / 128 GPUs respectively. For configuration A we also hold measurements out to 30 nodes (240 GPUs), cited where relevant.

Common settings: ioengine=libaio, direct=1, runtime=180, time_based=1, group_reporting=1

Throughput and IOPS figures in this post are aggregate sums of FIO results run simultaneously on all client nodes (FIO's "All clients" section). For example, 50 GiB/s at 16 nodes is the combined throughput of 16 machines running at once; divide by node count for a per-node average. Average latency is slat + clat (ms), and we present the 16-node result as the representative value.

Observations

Checkpoint restore — Sequential Read (1M)

This regime represents checkpoint restore and the linear loading of large training datasets.

• E and C lead. At 16 nodes, E (TCP 50 GiB/s, 40 ms latency) and C (RDMA 43 GiB/s) sit side by side at the front. E and B were already close to saturation at 8 nodes, C kept climbing through 16 nodes, and A stayed flat in the 22 GiB/s range from a single node onward.
• Convergence in the high-40 GiB/s range is a PoC link ceiling. That is close to the effective bandwidth limit of a 400 Gbps-class interface (roughly 46 GiB/s of payload), so the ceiling on C and E is not the storage itself but the measurement path. Larger builds have headroom left.
• D is an RDMA outlier. Throughput drops sharply on RDMA and latency rises to 140 ms — covered again in the outlier analysis in Observation 6.

A, 30-node measurement on hand: TCP 23.00 / RDMA 45.90 GiB/s — an unusual pattern where RDMA rises sharply from 16 to 30 nodes.

Checkpoint save — Sequential Write (1M)

This reflects the checkpoint-save regime.

• A leads alone. It saturates in the 46 GiB/s range from 8 nodes and holds flat out to 30 nodes. B comes close to A on RDMA but falls well behind on TCP.
• No configuration led on read and write at once. E, first in read, drops to the 11 GiB/s range in write, and D sits at 5–7 GiB/s with latency above 300 ms. In an environment where both checkpoint saves and restores are frequent, the two regimes have to be evaluated separately.
• A's RDMA is 17% below its TCP. Read on the same configuration is normal, so this looks like an outlier confined to the write path — re-analyzed in Observation 6.

A, 30-node measurement on hand: TCP 46.00 / RDMA 35.40 GiB/s. TCP flat, RDMA down slightly.

DataLoader — Random Read (4K)

This represents the small random I/O of an AI DataLoader. It is also the regime where the gap between configurations opened widest in this PoC.

• C is dominant. It enters the 3M IOPS range at 8 nodes already, with 16-node latency at 1.3 ms — high throughput and low latency at the same time.
• D shows the largest RDMA gain (+160%, a 2.6× increase). The RDMA effect grows steadily with node count, so for a random-read-dominated workload, RDMA must be enabled on D.
• E is indifferent to protocol choice. TCP and RDMA are effectively identical, with no gain from switching.

In large sample-based training where the DataLoader reads hundreds of thousands of small files at random, the IOPS ceiling in this regime maps directly onto perceived throughput.

A, 30-node measurement on hand: TCP 1,336 / RDMA 1,265 k IOPS.

Logs and metrics — Random Write (4K)

Characteristics of the log-writing, metric-recording, and intermediate-output regime.

• C leads again. It holds the 700k IOPS range from 8 nodes with the lowest latency, around 5 ms.
• Rankings flip with block size. E, near the bottom in sequential write, climbs into the upper-middle group at 370k in 4K random write.
• More nodes do not always help. A and D both have ranges where IOPS falls as nodes are added (A at 8 → 16 → 30 nodes: 192 → 174 → 147 k; D at 8 → 16 nodes: 288 → 210 k), so scale needs verifying where random write is a large share of the load.

A, 30-node measurement on hand: TCP 147 / RDMA 152 k IOPS.

Real DataLoader patterns — Multi-file Random Read (128K)

Mid-size chunked parallel reads, the closest match to an AI DataLoader's actual access pattern.

• The top four converge at a similar level. On 16-node RDMA, C, B, E, and A all land in the 41–49 GiB/s range. As with checkpoint restore, the link ceiling is a large factor.
• The largest RDMA gains appear here. B gains 77% on RDMA over TCP and A gains 48% — multi-file random read is the workload where switching to RDMA pays off most clearly.
• D is a scale-inversion outlier. It falls from 40 GiB/s at 8 nodes to 19 GiB/s at 16 nodes, less than half, with latency up more than fourfold. Same direction as the pattern in sequential read, analyzed together in Observation 6.

A, 30-node measurement on hand: TCP 23.00 / RDMA 53.40 GiB/s — RDMA rises once more in the 30-node range.

RDMA results and outlier analysis

RDMA reduces CPU involvement and copy overhead, so over the same link the expected direction is throughput equal to or higher than TCP. Change from TCP to RDMA at 16 nodes:

• Most results sit inside the expected range. B showed a consistent gain of +3% to +77% across every test, and E stayed within ±5%, meaning the two protocols are effectively equivalent. C also stays close to the ±5% noise band.
• Two conspicuous negatives are classified as outliers. They fall in parts of D and A, and are better read as tuning issues specific to a configuration-and-scale combination than as performance limits of the storage itself.

The ranges we classified as outliers:

• D — Seq Read −64%, Seq Write −26%, Multi-file RR −42%. Normal through 8 nodes, then a sharp drop only at 16 nodes. Latency at 16 nodes also rises considerably, to 140 ms on sequential read and 52 ms on multi-file random read. RDMA queues and congestion control are likely bottlenecks under concurrent multi-client access, so the stack needs tuning before scaling out.
• A — Seq Write −17%. Other tests on the same configuration are normal, so this looks confined to the write-commit and cache path rather than being a general RDMA-path problem.

The remaining small negatives (A random write −8%, C sequential write −6%, E's figures within −5%) we treated as measurement noise rather than outliers.

In short, RDMA behaved as expected in most configurations, and tuning-related inversions appeared only in specific configuration-and-scale combinations. The right way to read those outliers is as a signal to inspect the RDMA stack and scale parameters — not as a performance ceiling.

Configuration characteristics at a glance

• A — Strongest in sequential write, holding TCP flat in the 45 GiB/s range out to 30 nodes. RDMA gains on multi-file random read are also clear. Note the −17% outlier on sequential-write RDMA, which calls for a look at the write path, and that random-write IOPS declines gradually as nodes are added.
• B — The most consistent TCP → RDMA gain across every test (+3% to +77%). The configuration that benefits most in environments making active use of RDMA.
• C — Balanced across sequential, random, and mixed patterns. Secures high IOPS and low latency simultaneously on 4K random read and write.
• D — A large +160% RDMA gain on random read, but with RDMA throughput inversions (outliers) observed on 16-node sequential read and multi-file random read. Needs RDMA stack tuning before scaling out.
• E — Reaches close to the PoC link ceiling on sequential read (50 GiB/s at 16 nodes) with almost no TCP/RDMA difference, giving wide latitude in protocol choice. Saturation past 8 nodes is a link limit rather than the storage itself, so headroom in larger builds needs separate verification.

Metrics to check, by workload

Points worth referring to when applying these observations to real infrastructure design.

• Training dominated by a large DataLoader (images, multimodal, thousands to hundreds of millions of samples): the IOPS ceiling and latency of 4K random read and 128K mixed read
• Large LLM pretraining (frequent checkpoints of tens of gigabytes): how flat sequential write throughput stays as scale grows
• Restore- and reload-heavy operations (frequent training resumption, failure recovery): sequential read throughput and latency
• Moving to RDMA: validate in advance with FIO profiles close to the real workload

What this PoC confirmed is that choosing storage is not about picking the fastest product — it starts with defining which regime dominates in your own workload.

Drawing on the observations gathered here, Elice designs storage configurations together with you around training dataset size, checkpoint patterns, and scaling plans. From starting immediately on an ECI parallel file system to on-premises and hybrid builds based on AI PMDC, get in touch any time your AI training infrastructure is in question.