PVE9 New server bad performance NVME Gen5

[alex.76]

Hello everyone,

As part of a migration from VMWARE to Proxmox (with hardware replacement) of a Dell PE R7525 (which will be backed up), we purchased and assembled a server composed of:

SP5 SuperMicro H14SSL-N motherboard (last bios & BMC)
EPYC 9475F 48C / 96C HT processor
12x DDR5 ECC REG 6400
FSP Pro Twins 2x930w power supply
4x NVME Gen5 Enterprise (PLP) Kingston DC3000ME 7.86 TB (spec: https://www.kingston.com/en/ssd/dc3000me-data-centre-solid-state-drive? Capacity=7.68tb )
Cooling AIO + 4x Fan 140 Noctua IPPC + 3x Fan 120 Noctua IPPC + 5 Fan 80 Arctic P8 MAX
3x PCIe X710 Intel

The NVME are connected directly to the motherboard on 2 MCIOI8 ports 8x each (I tested the 3 present on the motherboard) and are negotiated in 4x each and Gen5.

First installation of Proxmox in RAID10 ZFS, I got quite disapting results from a Windows VM (fresh install). I spent a few hours trying to find suitable tweaks but without success....

I then installed Windows 11 directly on the server to pass the NVME individually to the Crystaldiskmark and the results on each of the NVMEs corresponded well to what could be expected.

I then reinstalled Proxmox by putting the OS on an M2 Onboard so that I could test on different scenarios.

VM configured on SCSI controller and I tried different configurations (No cache, writeback, directsync - natives, thread, uring....)

I also tested to modify ZFS by the way:

Direct=always
Primarycache=metadata
Atime=off

I also tried to create a ZFS volume with a single NVME and the results are also bad from the VM.

What I did to get the speed / IOPS results:

On the Windows VM, a Crystaldiskmark bench, the best I managed for the moment to get in ZFS RAID10 by putting the options: no cache / threads / IOthread ON:



On this capture, it is a benchmark from Win 11 installed on the server host directly (only one NVME no RAID):



I also launched an FIO from the Windows VM and I also took the opportunity to launch it from the host's zvol and here are the results:



To carry out the tests I used the configuration proposed by a member of the forum (thank to him) and available here: https://gist.github.com/NiceRath/f938a400e3b2c8997ff0556cced27b30

Only the ioengine has been modified on Windows by "=windowsaio"

Even if I had in mind that the ZFS was not totally performance-oriented, I really find that we are very low... As an example, a Crystaldiskmark launched from a VM of our current R7525 on RAID PERC card with 4x NVME Gen4 of 2021 is much more performer.

I must have missed something...

I am taking your advice to improve throughput / IOPS / Latency

Thank you for your help.

 

[_gabriel]

try with Lvmthin as ZFS has overhead, can't be compared to baremetal even more in benchmark.

Numbers should be closer if you run multi VM at same time, that's the point of virtualization.

For single VM :

D

Post in thread 'Improve virtio-blk device performance using iothread-vq-mapping'

Jun 10, 2025

I made patch that enables advanced iothread-vq-mapping for virtio-blk devices in Proxmox VE 8.4.

https://github.com/dominiaz/iothread-vq-mapping

Just add eg. iothread_vq_mapping=8 to your virtio drive configuration.

• dominiaz

• 

 

[Johannes S]

try with Lvmthin as ZFS has overhead, can't be compared to baremetal even more in benchmark.

But then he looses the other ZFS features:

[TUTORIAL] Thread 'FabU: This is just a small setup with limited resources and only a few disks, should I use ZFS at all?'

Jan 3, 2025

Should I use ZFS at all?​

For once this requires a disclaimer first: I am using ZFS nearly everywhere, where it is easily possible, though exceptions do exists. In any case I am definitely biased pro ZFS.

That said..., the correct answer is obviously: “Yes, of course” ;-)

Integrity​

ZFS assures integrity. It will deliver exactly the very same data when you read it as was written at some point of time in the past.

“But all filesystems do this, right?” Well, there is more to it. ZFS works hard to actively assure the correctness. An...

• UdoB
• consumer class enterprise class integrity mirror raidz storage zfs
• Replies: 1
• Forum: Proxmox VE: Installation and configuration

• 

 

[alexskysilk]

Why does any of this matter? does your use case require a specific throughput or is it just a matter of "thats what the specs say, so should the benchmarks!"

But then he looses the other ZFS features:

You cannot overstate this. a fast disk subsystem that can fail and take your data with it is useless. Never mind active checksum, inline compression, CoW features such as instant snapshot/recovery, etc.

The POINT of benchmarking isnt to show random numbers; those kind of benchmarks are of no use. What are you expecting to be doing? benchmark that.

 

[Kingneutron]

I'm a huge ZFS proponent, but it's been well-known for years that it has performance issues with nvme. They're working on it, but the code was written back when 500GB spinning disks were common.

If you want better performance with virtual disks, use lvm-thin. And make sure you have backups.

 

[alex.76]

try with Lvmthin as ZFS has overhead, can't be compared to baremetal even more in benchmark.

Numbers should be closer if you run multi VM at same time, that's the point of virtualization.

For single VM :

Thanks. I tried the patch but it didn't work. And after reading the topic, it supports up to the major release of the ISO for the moment.

I will try to see if it is only the ZFS that blocks or if there is something else. But in soft RAID I tend to favour ZFS. I did not foresee this in the plans but there is also the option to leave in hard Raid and eliminate ZFS.

Why does any of this matter? does your use case require a specific throughput or is it just a matter of "thats what the specs say, so should the benchmarks!"

Important I don't know, precisely it's a bit of my question. The purpose of the tests before starting the migration is to verify that everything is in order. CPU / RAM level, nothing to report (we are almost in 1:1 ratio between the host and the VM). On the other hand, when I see results and especially increases in latency and fairly low IOPS on the disk part and based on the same tests on our R7525 in HW RAID PERC and 4x NVME GEN4 where I have no increase in latency and much higher IOPS, I prefer to anticipate a possible problem (or not) and not find myself with problems on our VMs and especially those that are requested on MSSQL.

 

[alex.76]

I'm a huge ZFS proponent, but it's been well-known for years that it has performance issues with nvme. They're working on it, but the code was written back when 500GB spinning disks were common.

If you want better performance with virtual disks, use lvm-thin. And make sure you have backups.

If this type of result seems correct for ZFS on this typology, I did not foresee this but I may have to plan a transition migration to test that this does not influence the use compared to our current ESXi.

I can also consider starting again on HW RAID to ensure the blow.