ZFS Perfromance

[bolzerrr]

Hi,

i just made a fresh install with upgraded hardware (https://www.cpu-monkey.com/en/compare_cpu-intel_celeron_j4125-vs-intel_processor_n100, DDR4-3200 and a fast NVMKC3000 PCIe 4.0). On the former system i have a mdadm RAID-1 with 2 Sandisk value SSD drives and LVM on top of this.
Now i wanted to try ZFS Raid-1 using the NVM and a SSD on a SATA port as secondary:

zpool status
  pool: rpool
 state: ONLINE
config:

        NAME                                                           STATE     READ WRITE CKSUM
        rpool                                                          ONLINE       0     0     0
          mirror-0                                                     ONLINE       0     0     0
            nvme-eui.00000000000000000026b768660ec305-part3            ONLINE       0     0     0
            ata-SSD_2.5__512GB_InnovationIT_QLC_594322207150619-part3  ONLINE       0     0     0

I was hoping to get a good speed bump because of the NVM as primary disk, but the perfromance seems to be poor compared to the former MDADM RAID:

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.4408 s, 312 MB/s

MDADM:

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 1,21968 s, 880 MB/s

Is this expected?

 

[aaron]

There is no primary and secondary disk in a mirror. The slowest disk dictates the write speed.
Please do not use dd, especially if /dev/zero is the source for write benchmarks. Zeros are usually compressed very well. This tested basically how fast the CPU can compress the zeros before writing it down to disk. The fio command is much better suited.

ZFS does a lot more than an MD RAID, checksumming and Copy on Write for example. Therefore, don't expect to get better performance than with MD Raid. Plus, if you test with sync enabled, consumer SSDs can get quite slow once the cache is full. Not going to happen with just 1 GiB, but if you continue writing for longer (10 minutes), the performance will usually be reduced once the write cache is full.

Datacenter SSDs with power loss protection (PLP) will perform quite a bit better.

 

[tom]

See also:

https://pve.proxmox.com/wiki/Benchmarking_Storage

 

[bolzerrr]

Thank you for the hints.
The fio result are the other way round. I guess i will play a bit more with the VM to see how they perform - this is what is important in the end.

What would you recommend for a "consumer hardware" RAID-1? MD Raid, ZFS, other?

 

[LnxBil]

What would you recommend for a "consumer hardware" RAID-1? MD Raid, ZFS, other?

Depends on what you want:
- mdadm has no support and has to be maintained by you
- ZFS does have a lot of features that once grow accustomed to, you will miss badly (snapshots, incremental replication, Bit-rot detection, self healing ...)
- LVM (with ot without mdadm) is faster, yet lacks features.

 

[bolzerrr]

Thanks for the listing, i will stick with ZFS, it sounds very interesting.
This is now a bit off-topic, but can PVE use the ZFS snapshot functionality for the creation of backups?

 

[aaron]

This is now a bit off-topic, but can PVE use the ZFS snapshot functionality for the creation of backups?

Yes, and it is very useful for container backups. VM backups work differently and are storage type agnostic. VM backups create the snapshots on the virtualization level by starting to read the disk image, but when the VM wants to write some data to parts of the disk that haven't been backed up yet, those writes will be temporarily blocked. The disk areas in question are then backed up out of order. Once done, the write operation is allowed to continue. This way the disk image in the backup is exactly as it was at the time when the backup started.