Planning first installation on Dell R710 with 6x8TB drives

Stevan

New Member
Apr 12, 2023
1
0
1
I will be installing Proxmox 7.4 but this is a first with a raid setup. Looking to use Raid6 or Raid5 but my only experience is using mdadm to generate the array. Does anyone have a current best practices for 7.4 with raid6. TIA
My current setup on this hardware has Ubuntu Server running off a USB stick with the 3.5" HDDs as a main drive using Raid5 with mdadm.
Just looking for newer or best practices before starting the whole process.
 
Last edited:
I will be installing Proxmox 7.4 but this is a first with a raid setup. Looking to use Raid6 or Raid5 but my only experience is using mdadm to generate the array. Does anyone have a current best practices for 7.4 with raid6. TIA
My current setup on this hardware has Ubuntu Server running off a USB stick with the 3.5" HDDs as a main drive using Raid5 with mdadm.
Just looking for newer or best practices before starting the whole process.
Let ZFS handle it all for you.
 
I will be installing Proxmox 7.4 but this is a first with a raid setup.
I assume you mean "Hardware Raid with the built-in PERC"? Is this an "informed decision"? If yes --> go ahead (and use LVM etc).

If you want to use ZFS - which I highly recommend - you should flash your controller to "IT-mode" to let ZFS access the drives directly.

Regarding Raid6/Raid5: do not do this. You'll get only the IOPS of a single drive. For VM-storage always use a Raid10 equivalent.


Good luck :)
 
I assume you mean "Hardware Raid with the built-in PERC"? Is this an "informed decision"? If yes --> go ahead (and use LVM etc).

If you want to use ZFS - which I highly recommend - you should flash your controller to "IT-mode" to let ZFS access the drives directly.

Regarding Raid6/Raid5: do not do this. You'll get only the IOPS of a single drive. For VM-storage always use a Raid10 equivalent.


Good luck :)
What about the performance scenario when keeping raid 5/6 and using ZFS ZIL + L2ARC?
 
What about the performance scenario when keeping raid 5/6 and using ZFS ZIL + L2ARC?
Please stop this! :)
ZFS wants to access the disks directly; an HBA must be installed for this. You shouldn't do such wild mixes with RAID and ZIL and L2ARC either.

Many things may work, but they may not work well.
 
  • Like
Reactions: UdoB
Please stop this! :)
ZFS wants to access the disks directly; an HBA must be installed for this. You shouldn't do such wild mixes with RAID and ZIL and L2ARC either.

Many things may work, but they may not work well.
I was actually referring to what @UdoB was saying regarding RAID-10 vs Raid5/6 but with Zil + L2ARC... not with hardware raid.


To clear up my question:
If you are using an IT Flashed mode Controller and not a battery-backed Hardware RAID, can we still not get good performance with RAID 5 or RAID 6 if we use ZFS ZIL + L2ARC, when compared to RAID 10 with ZFS ZIL + L2ARC?

Thanks!
 
If you are using an IT Flashed mode Controller and not a battery-backed Hardware RAID, can we still not get good performance with RAID 5 or RAID 6 if we use ZFS ZIL + L2ARC, when compared to RAID 10 with ZFS ZIL + L2ARC?
A whole bunch of apples and oranges.

lets define a few things:
1. when you have a bunch of disk requests (eg, simultaneously from multiple vms) the more individual disk queues that are available to service those requests the better. In other words, striped mirrors would be more performant in a virtualization workload then parity raid. this is true for both hardware raid, software raid, zfs, etc.
2. ZIL and L2ARC are only useful in corner cases. for the vast majority of use cases they don't provide any benefit. In a system that doesnt have a lot of excess ram L2ARC can actually make things worse.
3. hardware raid can PERFORM better then zfs- but performance isnt the only thing of importance. zfs is a lot more robust as it provides checksums at all levels including file system, in addition to features such as in line compression, snapshot, snapshot replication/remote send, deduplication, etc. When you consider whats the most important aspect of a storage subsystem, I'd say performance is probably not the first.
 
A whole bunch of apples and oranges.

lets define a few things:
1. when you have a bunch of disk requests (eg, simultaneously from multiple vms) the more individual disk queues that are available to service those requests the better. In other words, striped mirrors would be more performant in a virtualization workload then parity raid. this is true for both hardware raid, software raid, zfs, etc.
2. ZIL and L2ARC are only useful in corner cases. for the vast majority of use cases they don't provide any benefit. In a system that doesnt have a lot of excess ram L2ARC can actually make things worse.
3. hardware raid can PERFORM better then zfs- but performance isnt the only thing of importance. zfs is a lot more robust as it provides checksums at all levels including file system, in addition to features such as in line compression, snapshot, snapshot replication/remote send, deduplication, etc. When you consider whats the most important aspect of a storage subsystem, I'd say performance is probably not the first.

1 - Make Sense
2 - Would you say Zil is better for VMs, as opposed to L2ARC?
3 - What about an Intel Optane as ZFS Zil Cache when compared to Hadware Raid?
 
@alexskysilk

Thanks for the resource. I will check it out.

I've got a Dell R620 with 8 x 1.2TB HDDs configured in RaidZ2.

This is the Benchmark that i've done:

Command:
Bash:
sync; dd if=/dev/zero of=tempfile bs=1M count=10240; sync

Results:
Bash:
Proxmox Host: ZFS /       =   1.5 GB/S
Proxmox VM on ZFS Datastore  (no Zil or L2ARC)   =   866 MB/s
 
It would be a pretty randomly used VM:
- Linux Web Server (httpd/nginx) application (Zabbix, nagios, nextcloud)
- DB (MySQL/PostGres/Mongo)
- Load Balancer (haproxy)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!