HW RAID or ZFS on Dell PowerEdge R630

[baritone]

Hi, I am wondering whether this is a correct usage of my hardware which runs PVE.

I have a Dell PowerEdge R630, 2x HDDs, and 2 SSDs. From what I understand, ZFS is not recommended to run on hardware RAID.
The controller (PERC H730 Mini) is currently set to HBA mode and I configured the Proxmox installation to use ZFS and mirror both HDDs. The SSDs are being used as VM storage and are ZFS mirrored also.

Would I achieve better results if I used hardware RAID on the H730 and ran Proxmox on ext4?
Could I also use HW RAID for the HDDs (boot) but ZFS for the SSDs?

I assume ZFS is better in terms of integrity and reliability but eats up a lot of memory.

I would like to hear your thoughts on my setup and would appreciate any advice.

 

[brucexx]

ZFS will eat more RAM , you could limit the use of RAM for ZFS but the RAM is what makes ZFS fast/faster then RAID. You could also do more with ZFS, replication Proxmox is one example. So out of the box ZFS is faster.

One of the disadvantages is that ZFS can cause excessive wear out on SSDs.

Thx

 

[waltar]

One of the disadvantages is that ZFS

One of the other disadvantages of zfs is that the users think zfs would be fast which isn't technically possible as it has more to do because of the checksums for both i/O ways but don't know better as don't use other filesystems too and so have no comparison against the truth.

 

[alexskysilk]

One of the other disadvantages of zfs is that the users think zfs would be fast which isn't technically possible

...

your "speed" isnt a function of any one thing. improperly tuned storage for the workload always performs poorly. I would say it is more accurate to say "a disadvantage of ALL disk aggregation mechanisms is that it requires the user to understand his workload prior to establishing disk aggregation policies."

zfs can and does perform well. and even if a "dumb" raid (not filesystem aware) has the potential to be faster its worth the tradeoff every time.

 

[waltar]

I like block checksums inside an external raid-system with dedicated hw resources for and a lightweight filesystem on top. Don't know how zfs could be called a zetabyte filesystem as it struggels in the tenth of terabytes, petabytes are just possible to use by combining multiple one with lustre.

 

[alexskysilk]

Don't know how zfs could be called a zetabyte filesystem as it struggels in the tenth of terabytes

lack of knowledge isnt a crime perhaps if you share your system and disk configuration along with your trouble experience others can offer help.

 

[waltar]

My trouble is just too fast systems and so conditioned for and everytime getting to a customer with zfs I could cry with the customer on I/O wait and problem is not to my side. And it looks you never restored 300 TB from backup after power outage to zfs pool as unfortunately DR servers for file and compute services are not the common mostly.

 

[alexskysilk]

And it looks you never restored 300 TB from backup

doing so as we speak and the number is actually AWEFUL close to the number I'm restoring. writing to a fresh zpool is really fast; they slow down as they cover 100% pool TBW as zfs does not have a defragmentation mechanism.

 

[UdoB]

My points pro ZFS: https://forum.proxmox.com/threads/l...-ssd-for-vm-ct-disk-images.156548/post-715473

 

[alexskysilk]

@UdoB you can add another point relevent to the above discussion (eg, large sets)- dcr. that backup is to restore a 10 year old filer that started dropping disks like flies, and I finally lost a vdev when it dropped disks during rebuild. all my new filers use dcr- which is a pretty rare feature on server raid controllers.

 

[UdoB]

you can add another point relevent to the above discussion (eg, large sets)- dcr

Excuse my ignorance... dcr=what? First hit in a search engine tells me "DatabaseChangeRegistration", but I have never heard of it...

 

[alexskysilk]

declustered raid (draid)

 

[UdoB]

declustered raid (draid)

Ah, ja! Das kam für mich bisher nicht in Frage, dazu bin ich sowohl im Dayjob als auch im Homelab zu klein

 

[alexskysilk]

als auch im Homelab zu klein

The man who moves a mountain begins by carrying away small stones. --Confucius

 

[baritone]

Thanks for the quick responses. I'll stay with ZFS. My other concern was whether turning the RAID controller to HBA mode would suffice for the "no hardware RAID" criteria?

 

[brucexx]

Depending on your setup on Dell servers you can simply setup drives to non-RAID on Perc controller. This allows for drives that you want to use with RAID to use RAID1, 6, 10 and the other drives will behave like JBOD. I found that setup to be most flexible.

 

[jdancer]

At work, I'm involved with migrating Dells from VMware to Proxmox.

I just make sure all hardware and firmware is the same (CPU, NIC, Storage, RAM, etc). I do swap out the PERC for Dell HBA330.

I use two small drives and use ZFS RAID-1 for Proxmox. Rest of drives are for VMs/data.

ZFS provides data and metadata error correction, snapshots, and compression. Win-Win-Win situation, IMO.