Proxmox VE ZFS Benchmark with NVMe

Dunuin

Famous Member
Jun 30, 2020
6,749
1,571
149
Germany
ZFS isn't that fast because it is doing much more and is more concerned about data integrity and therefore gets more overhead. With mdadm you don't get bit rot protection so your data can silently corrupt oder time, you get no block level compression, no deduplication, no replication for fast backups or HA, no snapshots, ...
You could add some of that features by using a filesystem like btrfs ontop of your mdadm array or using qcow images but than again your mdadm will get slower because the btrfs is creating additional overhead which you don'T get with ZFS because its all already integrated.
 

ectoplasmosis

New Member
Feb 2, 2022
7
0
1
42
As a comparison, I'm running a 15TB mdadm ext4 array on the following hardware:

EPYC 7443P 24-core CPU
256GB 8-channel RAM
8x Samsung PM9A3 Gen4 NVME U.2 1.92TB drives, connected via 2x AOC-SLG4-4E4T-O bifurcation re-timer cards

Seeing ~40GB/s read and ~20GB/s write via a sequential FIO test.
 

Emilien

Member
Jan 23, 2022
121
7
18
Italy
As a comparison, I'm running a 15TB mdadm ext4 array on the following hardware:

EPYC 7443P 24-core CPU
256GB 8-channel RAM
8x Samsung PM9A3 Gen4 NVME U.2 1.92TB drives, connected via 2x AOC-SLG4-4E4T-O bifurcation re-timer cards

Seeing ~40GB/s read and ~20GB/s write via a sequential FIO test.
Which mdadm raid config?
 

guletz

Famous Member
Apr 19, 2017
1,558
245
83
Brasov, Romania
With mdadm you don't get bit rot protection so your data can silently corrupt oder time
... even worst. If one of your HDD RAID10 member will have an non-fatal read error(any HDD has some read errors in time), then you will have a corrupt data/block. Even more, this read non-fatal errors(I can rea a block but I read other data compared with the data that was write in the past) will increase if your HDD temperature will be higher then the normal.... one night without your cooling system and you can lose all of your data.

Good luck / Bafta !
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!