PVE 6.3.6 - ZFS vs Local Directory FIO Performance on NVMe

[mpizzolo]

Hello Everyone,

I would be interested to understand whether anyone else has similar similar results, and what they've traced the issue to.

For evaluation purposes we have been testing some NVMe drives in a Proxmox cluster in both Local Drive and ZFS configuration (single disk, Micron 3.2TB 9300 MAX, rated at 3500 MB/s, 3100 MB/s; 835k iops, 310k iops for read and write respectively ). For ZFS we have tried both with and without compression and encryption.

At a high level the results as executed from a shell on the host: (AMD EPYC 2x 7542 32-core processor, 1TB RAM):


fio --filename=<STORAGE_PATH>/10G --size=10GB --direct=1 --rw=randrw --bs=8k --ioengine=libaio --iodepth=8 --runtime=60 --numjobs=4 --time_based --group_reporting --name=<STORAGE>_10G_8k_8_4X --eta-newline=1 --output-format=normal --output <STORAGE_PATH>_10G_8k_8_4X.txt

Results within the same storage destination vary 5%-10% from each other, but here is a representative middle of the pack result for each:

ZFS encrypted, uncompressed: Jobs: 4 (f=4): [m(4)][42.6%][r=145MiB/s,w=146MiB/s][r=18.6k,w=18.7k IOPS]
ZFS encrypted, compressed: Jobs: 4 (f=4): [m(4)][55.0%][r=145MiB/s,w=146MiB/s][r=18.5k,w=18.7k IOPS]
Data Disk w/ EXT4: Jobs: 4 (f=4): [m(4)][59.0%][r=822MiB/s,w=824MiB/s][r=105k,w=105k IOPS]

For relative comparison to another NVMe: Intel DC P4511 2TB rated at 2100 MB/s, 1430 MB/s; 295k iops, 36k iops:

Boot Disk EXT4: Jobs: 4 (f=4): [m(4)][48.3%][r=517MiB/s,w=518MiB/s][r=66.2k,w=66.3k IOPS]

The ZFS performance seems to be absolutely crippled at least as seen by fio using the parameters we chose to test with. We did try a few different iodepths, numjobs, and block sizes, but have not done exhaustive testing as yet. Before investing more time in this I wanted to see whether others had similar experiences.

Thanks in advance.

 

[0xd149e38e]

The ZFS performance seems to be absolutely crippled

Yes. ZFS encryption slowness is mind boggling.

Many people use zfs raid 1 and 10 with nvme because they want data safety and snapshots, not because they want performance.

Edit:
I found zfs on luks to be faster than zfs encryption.

PM983 m.2 zfs no encryption       : IOPS=16.8k, BW=131MiB/s
970 Evo + m.2 ext4 no encryption  : IOPS=58.2k, BW=455MiB/s 
970 Evo + m.2 xfs on luks         : IOPS=53.2k, BW=416MiB/s

It also depends a lot on the workload/benchmark. A few weeks ago we had a benchmark showing the P983 to be twice as fast as the 970 Evo.
https://forum.proxmox.com/threads/3...zfs-raidz1-poor-performance.87916/post-386543

 

[mpizzolo]

Thanks for your response.

To clarify though as I'm just realizing that I did not include the unencrypted speed, it is the SAME as the unencrypted speed.

I see ZERO material change between a ZFS disk with (or without) compression AND with (or without) encryption. All 4 combinations result in the same 150 MB/s range. It seems to be a ZFS issue and not a Compression or Encryption issue.

 

[0xd149e38e]

Well, you got very fast hardware, so encryption transactions won't harm your throughput, and you only got one drive.
So you figured out that ext4 is faster than zfs in your case.
Go ahead and use ext4. Proxmox supports it. What's the problem?

 

[Dunuin]

Go ahead and use ext4. Proxmox supports it. What's the problem?

If you buy such a expensive and powerfull server you might want high availability und reliability so you might want to use all the nice ZFS features that privide that but not ext4.

 

[mpizzolo]

We have many drives, and many servers, but the fact that the speed is crippled by a factor of 6x, 10x or more seemed like something worth reporting in case we were missing something, and/or in case someone else could benefit from the knowledge - or most importantly - might be able to verify and optimize for everyone's gain.

 

[0xd149e38e]

Explanation why zfs is not very fast:
https://www.reddit.com/r/zfs/comments/i0u9zs/horrendous_zfs_performance_on_nvme/

You still try some tuning:
https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html