New install on Dell R710 - Upgrade PERC 6 to H310 or H730?

[WhiteKnight976]

I'm currently looking to build a virtual environment for some certification studies. Specifically Palo Alto and Cisco equipment. So, it's not going to be a production environment really. I just finished installing Proxmax VE and ran into a few small issues. Upon looking into it, it appears I should have completed the initial and hypervisor config a bit different.

The server is a Dell PowerEdge R710 which is obviously dated. However, let me list the specs.

CPU: 2 × Intel Xeon L5640
144 GB RAM
Two 300 GIG SAS 15k drives (mirror controlled by PERC)
Two 2TB SATA 7.2k drives (mirror controlled by PERC)
PERC 6/i

So, after looking into it, it looks like it wasn't ideal to setup the mirror(s) via the PERC 6 controller.

What I've read online is this;
A PERC 6/i cannot do true HBA/IT mode. To use ZFS properly, you need to create individual, single-disk RAID 0 Virtual Drives for every hard drive so the controller passes them through to Proxmox.

The PERC 6/i s an older, legacy controller (SAS 6Gbps and limited to 2TB virtual disk sizes). If you are using it in a production environment, consider upgrading to an H200, H310 or H730 flashed to "IT Mode" or HBA Mode. This grants Proxmox direct, unhindered access to raw disks, which yields far better performance and reliability.

For those more informed than I, should I consider swapping the controller? I will probably have 2 virtual Palo FWs, 10 or Cisco devices and a handful of Windows 10 workstations when it's all setup.

Thanks!
Matt

 

[jdancer]

I have flashed a PERC 6 in a Dell R200 to IT-mode using Broadcom firmware using someone's web blog post. This was awhile ago.

12th-gen Dell PERC controllers (ie, H310) can be flashed to IT-mode per https://fohdeesha.com/docs/perc.html

I have these running production with no issues. Just don't skip any steps and take your time.

 

[J-Rod]

Since you are using hard drives, I think that the ability to use write-back caching with that RAID card (is the battery OK?) will be more beneficial than having ZFS handle the RAID. So just go ahead and set up ZFS on each of the mirrored arrays, and enable compression. When creating VMs, place the OS virtual disks on the 15k drives and larger, data storage virtual disks on the slower 7.2k drives.

If you were going to spend any money, I'd recommend buying a pair of Dell SSDs on eBay, and using those for the PVE boot array and the VM OS volumes. That will improve your quality of life quite a bit.

 

[CRCinAU]

I did this journey as well. Always go the H730 - as the write cache for HDDs gives a *MASSIVE* boost in performance.

 

[UdoB]

To use ZFS properly, you need to create individual, single-disk RAID 0 Virtual Drives

No!

Those disks, given to ZFS, are still virtual. They do not correspond to a physical disk. The Lackmustest is to query the SMART data for the drives ZFS see - if it works then you are possibly fine.

Theoretically you can create a ZFS pool on a Raid controller: use only one of such disks. Never add a second vdev. Never add any other thing like SLOG/Special Device. Having only a single physical device is the only chance to keep consistency on any crash/malfunction. You'll lose core features like self-healing as there is no redundancy on ZFS level with a single physical device. (Setting copies=2 probably is not a cure for this and it would reduce performance.)

Disclaimer: I am definitely not recommending this! In this world only a small part of possibilities is recommendable...

 

[J-Rod]

The OP is trying to make do with what they have, and they've said this is not production. Of course the RAID0 method is a hack of sorts.

If the goal is to learn about ZFS, then the RAID0 method or HBA replacement is the way to go, but I'd think performance will be poor with hard drives due to ZFS requiring writes to be fully written to disk before acknowledgement. So buy some SSDs instead of an HBA.

Even though it's ancient, the PERC 6/i does have 256MB of onboard cache memory and supports a battery backup unit, so the write acknowledgement once the data hits cache will be a performance boost. So if the goal is just to get a bunch of VMs up and running, and as performant as possible, then the hardware RAID configuration seems ideal to me.

What filesystem to choose for this specific setup might be a further debate and if you do some searching you can see that of course ZFS does lose out in some situations, but in this case I'd recommend it so that you can use compression. 300GB is going to be tight for all of those VMs, and compression will save a lot of space in this situation. It can also help to reduce the I/O load on the underlying disks.

 

[UdoB]

Sure. If it works for @WhiteKnight976 then I am fine

I just wanted to make sure he understands the implications of using a possibly risky approach. Better know that from the beginning than suffering later...

 

[J-Rod]

There are various risk factors here. I'd say user error while learning is probably near the top, then the single host next. I don't see the PERC6 or ZFS with RAID0 as much of a risk by comparison. Data loss risk could be partially mitigated by running a PBS VM on the 2TB mirror and doing regular backups. A better option in my opinion would be to find a 2nd machine, even an old desktop, install PVE on it, then PBS as a VM. Now you have a backup of your main server VMs and can even live restore and launch a VM or two while you fix the R710.

Side node: of course a top choice is a three-node cluster of some sort using newer hardware, although the 144GB of RAM in the R710 is going to be tough to replicate if your budget is small. But there is some help for this. I have a couple three-node clusters and have had a better experience with memory overcommit (primarily during maintenance tasks) by making some configuration tweaks. Of course enterprise SSDs with ZFS mirror for boot and VMs, but leave some space during install available for swap via LVM mirror (50% of RAM is my go-to). The primary reason for needing swap is due to the fact that both KSM and ballooning take some time to run, especially during incoming VM migrations, so the host will likely kill some VMs early even if all the VMs could easily fit eventually. Also, PVE defaults to KSM and ballooning both kicking in at 80%, so I recommend setting ballooning to 85% or 90% so KSM is leveraged first. With this arrangement, I can run all three hosts at 80% memory utilization normally, but during maintenance can still fit all the VMs on two nodes. More info available from your favorite YouTubers. However, this is probably not recommended for critical production use because the swap space becomes an additional failure point (triple mirror?).

 

[UdoB]

install PVE on it, then PBS as a VM.

Yes, it works. Actually I use that construct too! But I wouldn't recommend it. (As we know that PBS runs best when it has its own physical storage. And K.I.S.S.)

run all three hosts at 80% memory utilization normally, but during maintenance can still fit all the VMs on two nodes.

Cool! Nevertheless I feel that's a... brave approach

 

[J-Rod]

To further clarify/expand some of my previous statements:

Everything as a VM has been my very successful mantra for over two decades. I'm for sure not going backwards just for PBS, especially when it offers essentially no benefit as bare metal and a very long list of advantages as a VM.

Just under 80% memory utilization is a common PVE result since that is where KSM and ballooning stop additional work. One of the major points of virtualization is to increase consolidation/utilization, so unused RAM is just a waste of money in my opinion. I'd prefer to let the VMs use it. Also, be sure to set the minimum memory sanely when using ballooning.

There are plenty of forum posts by users getting caught out by PVE unceremoniously killing their VMs when memory runs low (especially on ZFS, since this requires additional steps). It's perhaps a slightly less common issue with large enterprise deployments where 20% free memory might be hundreds of gigabytes. However, I'd always recommend leaving some free disk space when installing PVE on ZFS, so you can create swap later. And once set up, you can easily turn swap off and on to suit various conditions.