Planning first installation on Dell R710 with 6x8TB drives

[Stevan]

I will be installing Proxmox 7.4 but this is a first with a raid setup. Looking to use Raid6 or Raid5 but my only experience is using mdadm to generate the array. Does anyone have a current best practices for 7.4 with raid6. TIA
My current setup on this hardware has Ubuntu Server running off a USB stick with the 3.5" HDDs as a main drive using Raid5 with mdadm.
Just looking for newer or best practices before starting the whole process.

 

[somniture]

I will be installing Proxmox 7.4 but this is a first with a raid setup. Looking to use Raid6 or Raid5 but my only experience is using mdadm to generate the array. Does anyone have a current best practices for 7.4 with raid6. TIA
My current setup on this hardware has Ubuntu Server running off a USB stick with the 3.5" HDDs as a main drive using Raid5 with mdadm.
Just looking for newer or best practices before starting the whole process.

Let ZFS handle it all for you.

 

[UdoB]

I will be installing Proxmox 7.4 but this is a first with a raid setup.

I assume you mean "Hardware Raid with the built-in PERC"? Is this an "informed decision"? If yes --> go ahead (and use LVM etc).

If you want to use ZFS - which I highly recommend - you should flash your controller to "IT-mode" to let ZFS access the drives directly.

Regarding Raid6/Raid5: do not do this. You'll get only the IOPS of a single drive. For VM-storage always use a Raid10 equivalent.


Good luck

 

[Zubin Singh Parihar]

I assume you mean "Hardware Raid with the built-in PERC"? Is this an "informed decision"? If yes --> go ahead (and use LVM etc).

If you want to use ZFS - which I highly recommend - you should flash your controller to "IT-mode" to let ZFS access the drives directly.

Regarding Raid6/Raid5: do not do this. You'll get only the IOPS of a single drive. For VM-storage always use a Raid10 equivalent.


Good luck

What about the performance scenario when keeping raid 5/6 and using ZFS ZIL + L2ARC?

 

[sb-jw]

What about the performance scenario when keeping raid 5/6 and using ZFS ZIL + L2ARC?

Please stop this!
ZFS wants to access the disks directly; an HBA must be installed for this. You shouldn't do such wild mixes with RAID and ZIL and L2ARC either.

Many things may work, but they may not work well.

 

[Zubin Singh Parihar]

Please stop this!
ZFS wants to access the disks directly; an HBA must be installed for this. You shouldn't do such wild mixes with RAID and ZIL and L2ARC either.

Many things may work, but they may not work well.

I was actually referring to what @UdoB was saying regarding RAID-10 vs Raid5/6 but with Zil + L2ARC... not with hardware raid.


To clear up my question:
If you are using an IT Flashed mode Controller and not a battery-backed Hardware RAID, can we still not get good performance with RAID 5 or RAID 6 if we use ZFS ZIL + L2ARC, when compared to RAID 10 with ZFS ZIL + L2ARC?

Thanks!

 

[alexskysilk]

If you are using an IT Flashed mode Controller and not a battery-backed Hardware RAID, can we still not get good performance with RAID 5 or RAID 6 if we use ZFS ZIL + L2ARC, when compared to RAID 10 with ZFS ZIL + L2ARC?

A whole bunch of apples and oranges.

lets define a few things:
1. when you have a bunch of disk requests (eg, simultaneously from multiple vms) the more individual disk queues that are available to service those requests the better. In other words, striped mirrors would be more performant in a virtualization workload then parity raid. this is true for both hardware raid, software raid, zfs, etc.
2. ZIL and L2ARC are only useful in corner cases. for the vast majority of use cases they don't provide any benefit. In a system that doesnt have a lot of excess ram L2ARC can actually make things worse.
3. hardware raid can PERFORM better then zfs- but performance isnt the only thing of importance. zfs is a lot more robust as it provides checksums at all levels including file system, in addition to features such as in line compression, snapshot, snapshot replication/remote send, deduplication, etc. When you consider whats the most important aspect of a storage subsystem, I'd say performance is probably not the first.

 

[Zubin Singh Parihar]

A whole bunch of apples and oranges.

lets define a few things:
1. when you have a bunch of disk requests (eg, simultaneously from multiple vms) the more individual disk queues that are available to service those requests the better. In other words, striped mirrors would be more performant in a virtualization workload then parity raid. this is true for both hardware raid, software raid, zfs, etc.
2. ZIL and L2ARC are only useful in corner cases. for the vast majority of use cases they don't provide any benefit. In a system that doesnt have a lot of excess ram L2ARC can actually make things worse.
3. hardware raid can PERFORM better then zfs- but performance isnt the only thing of importance. zfs is a lot more robust as it provides checksums at all levels including file system, in addition to features such as in line compression, snapshot, snapshot replication/remote send, deduplication, etc. When you consider whats the most important aspect of a storage subsystem, I'd say performance is probably not the first.

1 - Make Sense
2 - Would you say Zil is better for VMs, as opposed to L2ARC?
3 - What about an Intel Optane as ZFS Zil Cache when compared to Hadware Raid?

 

[alexskysilk]

2 - Would you say Zil is better for VMs, as opposed to L2ARC?

search the forums for more specific data. this may be of help: https://forum.proxmox.com/threads/zfs-tests-and-optimization-zil-slog-l2arc-special-device.67147/

But in all honesty, if you're deploying ssds you'd be better served with an ssd pool instead. HDDs are SO. MUCH. SLOWER. in almost everything, they're really only useful for bulk storage.

 

[Zubin Singh Parihar]

@alexskysilk

Thanks for the resource. I will check it out.

I've got a Dell R620 with 8 x 1.2TB HDDs configured in RaidZ2.

This is the Benchmark that i've done:

Command:

sync; dd if=/dev/zero of=tempfile bs=1M count=10240; sync

Results:

Proxmox Host: ZFS /       =   1.5 GB/S
Proxmox VM on ZFS Datastore  (no Zil or L2ARC)   =   866 MB/s

 

[alexskysilk]

sync; dd if=/dev/zero of=tempfile bs=1M count=10240; sync

What workload is this meant to be simulating? all this shows is how fast you can shove zeros to the buffer.

 

[Zubin Singh Parihar]

It would be a pretty randomly used VM:
- Linux Web Server (httpd/nginx) application (Zabbix, nagios, nextcloud)
- DB (MySQL/PostGres/Mongo)
- Load Balancer (haproxy)