Slow NVME Performance on ZFS Mirror

[jafilip]

Hello, I've been searching the forums for days now trying to find a suitable solution to my increasingly slow NVME drive performance. Here's the details on the server:

1. ASRock EP2C602 Dual socket motherboard
2. 2x Intel 2560v2 CPUs
3. 128Gb ECC registered memory
4. LSI HBA (card passthrough to VM)
5. 8x Ironwolf HDDs (Storage server)
6. 2x Kingston 250Gb SSD ZFS Mirror (Proxmox host)
7. 2x Samsung 970 EVO 1Tb NVME ZFS Mirror (VMs hosted on here)
8. 2x PCIE to M.2 adapter cards (for NVME drives)
9. AMD RX 470 (Passthrough to Windows VM)
10. Proxmox 7.3

Testing drive speeds while in the ZFS Mirror configuration, I'm seeing speeds on both drives (tested each drive separately) of 200MiB/s using fio commands, at 4k block sizes. Inside the Windows 10 VM, using Crystal DiskMark, I'm seeing results of 500MB/s read and writes (which makes sense given the results from Proxmox). I've done my due diligence and even checked Link speeds and capacity on PCIE adapters to make sure I'm seeing the correct speeds and both drives are reporting PCIE 3x4 speeds. Mind you, Samsung claims speeds in excess of 3000MB/s, so I'm seeing only a fraction of those speeds in the real world. Having done some research, I've come to the conclusion that the drives are performing so poorly because they're not designed for the high write amplification due to ZFS. Am I on to something here? I'm curious to try and pull one of the drives off the pool, reformat it ext4 and do another test to see if it performs even marginally better, but that would require time and introduce possible data corruption if the other drive fails during my testing.

Anyone have any useful tips/tricks that can ho tune ZFS for some better performance? I've tried disabling zfs_sync, but would prefer data integrity over drive performance.

Maybe the adapter card is to blame? Cheap cars off of Amazon. Wouldn't think it would bottleneck the drive this badly if it's just converting PCIE x16 to x4...should have little to no impact on performance.

Basically stumped at this point. Any help would go a long way.

 

[apoc]

2x Samsung 970 EVO 1Tb NVME ZFS

Imho. You are bringing a knife to a gunfight.
Consumer ssds and zfs are not playing nicely together. ZFS was built for datacenter usage. Consumer ssds are built for, well, consumers...

Zfs has a huge write amplification through the journal approach. Basically every write is multiplied.
There are tons of posts explaining this in detail. Bottom line I think is: your tried to save money (and failed).

Aside that Storage performance in general is a tough topic. Don't believe the numbers someone gives you. Steady state is far less. In virtualization one individual worker might not see best results either...

 

[jafilip]

What makes Datacenter SSD better for ZFS? Does it have to do with caching on the drive itself? Just trying to look out for a good drive that performs admirably. Just need to know what specs I should be looking out for.

Also, budget wasn't a factor in this decision. Speed and endurance were. I had better performance on Truenas using the same drives, but inevitably moved to Proxmox for more versatility and flexibility. Figured Samsung's lineup was sufficient for what I needed in terms of speed and was not aware of the ramifications ZFS might have on the drives.

 

[jafilip]

Imho. You are bringing a knife to a gunfight.
Consumer ssds and zfs are not playing nicely together. ZFS was built for datacenter usage. Consumer ssds are built for, well, consumers...

Zfs has a huge write amplification through the journal approach. Basically every write is multiplied.
There are tons of posts explaining this in detail. Bottom line I think is: your tried to save money (and failed).

Aside that Storage performance in general is a tough topic. Don't believe the numbers someone gives you. Steady state is far less. In virtualization one individual worker might not see best results either...

Also, haven't you seen John Wick? You can do some real damage with a knife .

 

[apoc]

What makes Datacenter SSD better for ZFS? Does it have to do with caching on the drive itself? Just trying to look out for a good drive that performs admirably. Just need to know what specs I should be looking out for.

It starts with the fact that wear-leveling is (at least for the mixed use SSDs) working different. DC SSDs are built for steady state performance, not peak-performance as it is typically done for consumer SSD.
Due to the nature of ZFS the SSDs constantly receiving writes. From my perspective this leads to the consumer SSD falling into steady state performance which is typically a fraction of the peak performance.

Vendors usually claim optimal performance values, like bandwidth figures of the physical connection etc.

I had better performance on Truenas using the same drives

maybe the software is a factor here as well. In most cases I have read the issue though is the fact that the used gear is not up to the job.
There are tons of other factors which might play a role here. Storage Performance and troubleshooting is no piece of cake, starting with the understanding what is actually measured.

Also, haven't you seen John Wick? You can do some real damage with a knife .

you have asked for an opinion / ideas. I gave you my point of view and tried to point out an issue.
If you don't like what you read - that's you issue.

 

[Dunuin]

What makes Datacenter SSD better for ZFS? Does it have to do with caching on the drive itself?

In addition to what apoc wrote.
Power-loss Protection allow enterprise SSDs to cache sync writes in DRAM cache, giving you magnitudes of better sync write performance. The spare area is bigger so more space to use as SLC-cache, to replace failed NAND cells increasing life expectation and more space to move data around for optimization, even if that disk is pretty full. A 960GB consumer SSD acrually got 1024GB with 64GB used as spare area. A 800GB enterprise SSD also got 1024GB but 224GB spare area.
And often better flash chips (MLC instead of TLC or TLC instead of QLC).

 

[samsausage]

Hello, I've been searching the forums for days now trying to find a suitable solution to my increasingly slow NVME drive performance.

What is the ashift on your ZFS Pool? If it is set too low, you may be experiencing write amplification.

 

[jafilip]

What is the ashift on your ZFS Pool? If it is set too low, you may be experiencing write amplification.

Just checked. Ashift size = 12. Other recommend 13 for the NVME drives I've currently using, but going through the process of changing that would be negligible in terms of gained speeds.