Hello,
I'm benchmarking different storage setups on a new server and I've found an issue or strange behaviour.
The server has 4x KIOXIA KCMYXRUG3T84 on AMD EPYC, linking as expected on PCIe 5.
Running a single windows server VM on single disk reports reads about ~15GB/s, that is as expected for that drive.
Clonning that machine on to 4 disks and running the same test simultaneously, reports 15GB/s for every disk. nmon reports about aggregated 55GB/s at the hypervisor.
But, if I combine the 4 disks into any kind of RAID (mdadm, tested levels 0, 1, 10, 5), ZFS RAIDZ, or LVM, or combiations of them, even in directory mode, the read speed is always equal or below 14GB/s. It seems that it is a kernel issue, or setup limitation, but until now, I haven't found the cause of the bottleneck.
For example, mdadm 4 drive raid 0 (tested even setting 1k chunk) mounted as directory over ext4, with the test VM on it, exactly as if I partition a single disk, gives 15GB/s.
The disk is set as IOThread, No cache, Async IO: Native
Any research idea to achieve optimal RAID or LVM performance?
Current results:
Thanks!
I'm benchmarking different storage setups on a new server and I've found an issue or strange behaviour.
The server has 4x KIOXIA KCMYXRUG3T84 on AMD EPYC, linking as expected on PCIe 5.
Running a single windows server VM on single disk reports reads about ~15GB/s, that is as expected for that drive.
Clonning that machine on to 4 disks and running the same test simultaneously, reports 15GB/s for every disk. nmon reports about aggregated 55GB/s at the hypervisor.
But, if I combine the 4 disks into any kind of RAID (mdadm, tested levels 0, 1, 10, 5), ZFS RAIDZ, or LVM, or combiations of them, even in directory mode, the read speed is always equal or below 14GB/s. It seems that it is a kernel issue, or setup limitation, but until now, I haven't found the cause of the bottleneck.
For example, mdadm 4 drive raid 0 (tested even setting 1k chunk) mounted as directory over ext4, with the test VM on it, exactly as if I partition a single disk, gives 15GB/s.
The disk is set as IOThread, No cache, Async IO: Native
Any research idea to achieve optimal RAID or LVM performance?
Current results:
| CrystalDiskMark test | Read (MB/s) | Write (MB/s) |
| RAID 10 + LVM | 13884 | 10182 |
| RAID 10 + LVM Thin | 7910 | 3049 |
| RAID 1 + LVM | 13131 | 6515 |
| RAID 5 + Directory | 1129 | 1106 |
| ZFS mirror | 7996 | 7765 |
| ZFS RAID 10 | 7648 | 11445 |
| ZFS RAIDZ | 7915 | 11714 |
| ZFS RAIDZ2 | 7918 | 6191 |
| ZFS dRAID (2 DATA 1 SPARE) | 7623 | 9542 |
| ZFS on single disk | 7699 | 8682 |
| ZFS on mdadm RAID0 | 7872 | 10338 |
| ext4 on RAID0 Directory | 16063 | 13764 |
| ext4 on RAID1 Directory | 15481 | 6120 |
| ext4 on single disk Directory | 14962 | 7580 |
Thanks!
Last edited: