Proxmox 6.2 on ZFS on HPE Proliant servers

[macleod]

Topic: Proxmox 6.2 on ZFS (live migration included) on HPE Proliant servers. (Sorry for such a long post )
ceph is not quite a solution, because it requires (or at least highly recommends) at least 10gigs network

I have some older and some newer hp servers, like:
- HP Proliant DL380 G6 / Smart Array P410i (old piece of $h1t, for testlab or smth)
- HP Proliant DL360 Gen8 / Smart Array P420i
- HP Proliant DL360 Gen10 / HPE Smart Array P408i-a SR Gen10

- HP Proliant DL380 G6 / Smart Array P410i / storage solutions:
1. lvm-thin
- PROs: hot swap, official supported
- CONs: go shoot yourself with live migration (thin becomes thick - as least in migration time); start migrate and go to coffee, lunch, maybe dinner ; possible bitrot problems (will remain undetected)
2. JBOD or IT mode is not supported on P410i (unless itanium), so ... it's a no-no
3. zfs raid0 over hardware raid (i.e. raid6 onboard)
- PROs: hot swap for disks, array will get rebuild automatically
- CONs: probably the worst solution, single bit rot and zfs will go crazy as zfs cannot correct data (only realize mismatched checksum)
4. zfs raidz2 over N x hardware raid0
- PROs: using "what you already have"
- CONs: no hot swap, you must reboot when change failed drive (or at least re-create raid0 array with new disk, remove old raid0, add new raid0 dev to zfs)
5. zfs raidz2 over disks connected to some extra HBA board
- PROs: best version for zfs (direct access to disks)
- CONs: need to buy new hardware; also new data cables; if new board is not hp branded server fans will go crazy

Questions:
frankly I should go with 4th (raidz2 over N x hardware raid0, like I've said not optimal solution, but with zero investement in an already archaic server
... did anyone tested this scenario and answer to this questions before I can get some tests:
Q1) can the server boot "as is" from such configuration or I must also add some raid1 for the system partition
Q2) how painful it is for zfs when a hard is malfunction ? as far as I read zfs will see only read errors from that drive and eventually will kick out the drive for that reason
Q3) did anyone has problems changing failed hard drives or at the very next boot (when disks we're changed "online")?
Q4) also, can anyone recommend some hba board for 5th case ? (so the fans will not go mad and the zfs recovery for failed drives will go easier?)

- HP Proliant DL360 Gen8 / Smart Array P420i
1. lvm-thin: as above, not optimal for live migration
2. JBOD is supported, but AFAIK you cannot boot from non-raid drive, so you need to install proxmox on some ssd board or to use 2 bays to create a raid1 installation disk (assuming mixed mode - both raid and jbod - to be supported)
3. zfs raid0 over hardware raid (i.e. raid6 onboard); like I've said before, worst solution possible, silent errors will turn into disaster

I've just read that mixed mode (JBOD and RAID) is not (yet) supported on P420i, so best solution seems to be that system (or at least /boot) partition to be installed / moved to an onboard flash drive. Any better solution ?

- HP Proliant DL360 Gen10 / HPE Smart Array P408i-a SR Gen10
must disable both secure boot and uefi (although debian 10 will boot as secure, proxmox installed won't; also surely it will not boot securely after "magical" install); so ... boot in legacy mode
they say you can use at least 4 disks in IT mode to create a zfs raidz2 array and the system should boot; can anyone confirm ?

Thanks! (and again sorry for such a long "story")

 

[aaron]

If I understand you correctly, you would like to create a cluster out of a mix of G6, G8, and G10 HP servers right?

Ceph is out of the question and you would like to use ZFS?

I have some personal experience with G6 and G8 servers. To be honest, go ahead, buy some H220 HBAs and a set of cables with straight plugs for the G8 and switch the backplane over to the HBA, leaving the RAID controller unused. Fiddling around with Raid0 disks underneath ZFS will be a PITA sooner or later.

In my personal experience, the H220 works fine on a G6 even though the side channel is not compatible and thus the LEDs on the HDDs won't work.
The cable with straight plugs is needed for the G8 AFAIR because the plugs on the onboard RAID controller show up straight and the installed cable uses plugs turned 90 degrees which will not fit on the HBA.

 

[macleod]

If I understand you correctly, you would like to create a cluster out of a mix of G6, G8, and G10 HP servers right?

Not necessarily mixing ancient and new in the same cluster, but putting some hardware to work ...
As I looked for the G6 specs, only HPE SC08Ge Host Bus Adapter is listed as "compatible" HBA card ... and I know HPE for being very picky in terms of compatibility, even "it seems to work"
Thanks for the feedback!

 

[yurtesen]

@macleod for "3. zfs raid0 over hardware raid (i.e. raid6 onboard)" you can resolve your dilemma using `copies=2`
See: https://docs.freebsd.org/en/books/handbook/zfs/#zfs-quickstart
You also get use of the raid cache memory (assuming you have battery backup it is safe).
On top of it, it is easy to determine which drive is failed using the drive leds. Not sure how one can do it easily if HBA mode is used...

About " 5. zfs raidz2 over disks connected to some extra HBA board" You should upgrade to smartarra p440ar for example. It can be switched to HBA mode and it costs ~60euro on eBay....

 

[leancode]

We are running 3 Gen7 DL360 in production with 8xraid0 on the P410i and full ZFS as raidZ2. All works fine and as expected. We are running 7 disks as RAIDZ2 and one SSD for the logs and cache. We recently needed to replace one disk and yes it requires a reboot because the P410i disables the disk when you pull it out and does not re-enable it when you plug in the new one. This could possibly be done with the cli utility but from my understanding you would have to remove the drive and then re-add it and I read a warning somewhere that this will change drive assignments so I choose to go the reboot route and it went fine. But I really should test it once as a test with the cli to see if it works.

 

[benoitc]

old thread but I wonder why not using disks in LBA mode with the P408i-a SR Gen10:
https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008200enw

Mixed Mode (RAID logic drives and HBA physical drives simultaneously)

 

[benoitc]

old thread but I wonder why not using disks in LBA mode with the P408i-a SR Gen10:
https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008200enw

anyone tested it already? I hesitate between thid or judt using ceph and use 2 disk with the hw raid. thoughts?