Skip to main content
These are orders of magnitude, not a leaderboard. Every figure on this page comes from the same benchmark, run the same way, so the rows can sit side by side — but they come from different clusters with different hosts and networks, so read them as what each storage class delivers as deployed here, not as a verdict on the media alone.

​
How it was measured

One recipe everywhere — the platform’s e2e disk benchmark, identical in Linux VMs, in pods, and (via DiskSpd with the same parameters) in Windows VMs: At a queue depth of 16 with one job, the IOPS figure is simply 16 divided by the average latency — the queue never fills. So these numbers describe how long one I/O takes, which is what an application at that depth feels, and not the maximum a drive can be pushed to. A datasheet figure is measured at a far higher depth and is not comparable with anything here. The pre-fill matters more than it looks, and it can be wrong in either direction. On a thin-provisioned array a read of a region never written measured more than 5× slower than the same read after one write pass; on ext4, a direct read of an unwritten extent is answered with zeros and never touches the disk at all.

​
In a VM

Measured 2026-09-17 to 2026-09-26, on quiet clusters (no replication resync running). Rounded to the order that carries information. What this says:
  • Replicated local NVMe on enterprise drives and the Fibre Channel array are the same order of performance inside a VM. Writes land within a few percent of each other; the replicated class reads somewhat faster and has the lower write tail.
  • Every replicated write pays a network round trip, and so does vSAN. On vSphere the same VM wrote about half as fast on a two-host vSAN mirror as on a single local datastore on the same host — the mirror waits for the other host. On these clusters the KubeVirt replicated class wrote about twice as fast as the vSAN mirror; the hosts and networks differ, so that ratio is a property of the two deployments, not of the software alone.
  • A single consumer drive with no replica is fast. The local VMFS row is a consumer NVMe on its own, and it writes as fast as anything above. What it lacks is not speed — it is power-loss protection and endurance (a fraction of an enterprise drive’s rated writes), and the next section is what happens when you ask it to replicate.

​
In a pod

The same fio command in a pod on the same class, with no virtual disk in the path: Without the VM layer the array’s write cache shows: it acknowledges writes faster than two NVMe drives and a network round trip can. Inside a VM the virtual disk path is the narrower part, and the two classes meet.

​
Consumer NVMe under replication — information only

The same pod benchmark on linstor-drbd with two copies on consumer NVMe (Samsung 990 PRO, no power-loss protection): That is roughly fifty times fewer writes than the same class on enterprise drives, with a tail two orders of magnitude longer — while the reads are the fastest of any class measured. It is also not stable: the same class on the same drives, at a deeper queue, has measured anywhere from about 150k to about 27k writes on different days. The drive’s internal state decides, not the platform. Treat these figures as an illustration, never as a baseline. The mechanism is the reason the drive class matters and not just the network. DRBD — like any synchronous replication — acknowledges a write only once the peer has it on stable storage, and it issues cache flushes to make sure of that. An enterprise drive with power-loss protection completes a flush from protected cache; a consumer drive has to put the data on flash, every write, on both nodes. Reads never flush, which is why the same drives look excellent on reads and unusable on replicated writes.
Replicated local storage requires enterprise NVMe with power-loss protection. Consumer drives are acceptable for single-copy pools and for data you can afford to lose. See LINSTOR.

​
Reading your own numbers

  • Compare only like with like — the same queue depth, job count, file size, pre-fill, and the same form (VM or pod). A VM and a pod on the same class gave different answers above; a benchmark at queue depth 32 is not comparable with one at 16.
  • Check for a resync before you measure. A replicated volume that is still copying to its peer measures about half its steady-state rate, and a freshly cloned VM disk is exactly that.
  • Check which node you landed on. On one supervisor one node of three serves the array at half the rate of the others for reasons not yet found; a benchmark that happens to land there reports half the figure.
  • Warm a consumer drive before trusting it. A single run on a consumer NVMe reports wherever its write cache happened to be.

​
Reproduce it

In a pod, from the platform. Celum runs this exact recipe in a short-lived pod on any StorageClass and returns fio’s own numbers — including fio’s job options, so a result states what actually ran: The recipe is fixed; you choose the class, optionally the node, and which of randread, randwrite, read, write to run. One benchmark runs per supervisor at a time — a second request is refused with the id of the one running, because two benchmarks on one array measure each other. The 10 GiB claim and the pod (namespace celum-benchmark) are deleted afterwards. It needs cluster-agent 1.141 or newer, whose image ships fio; an older agent is refused with that reason. By hand. In a Linux VM or a pod with fio, on a filesystem on the class under test:

LINSTOR

The replicated and shared pools, and which class a workload wants.

Storage overview

The backend catalogue, and how to choose a family.