Proxmox 7.1 won't boot on ZFS and 440AR in HBA mode (HP DL380 Gen9 server)

[Xealo]

Hello,

I want to install Proxmox 7.1 on HP DL380 Gen9 with UEFI Mode. I managed to switch the HP Smart array P440ar controller to HBA mode. The goal is to setup the 4x 10 TB Seagate drives in ZFS RAID10 mode.

First I tried the ISO installer in UEFI boot mode. The installation seemed to succeed.

The problem is that once Proxmox 7.1 is installed and the server is restarted... the server does not start on the "Proxmox" installation because it does not find the bootloader and asks for bootable devices. Trying the different disks does not remedy the situation.

I have seen this similar thread and have the same problems with Proxmox V7.1 iso 2 and have tried the described manual procedure but with no success.

Any advice? Thanks a lot!

 

[bobmc]

I think you're asking too much of the hardware you have.
1. The P440ar in 'HBA' mode is not a good match for a ZFS Pool - yes, it will 'work' but zfs works best (and more importantly for your data, more reliably) when it has direct access to the drives - which it doesn't really have on a controller like the P440ar.
2. The Gen9 systems were one of the earlier systems to support UEFI and you may have more luck with legacy mode

I'd advise you to consider replacing the P440ar with an LSI card, you may have to replace the two backplane cables as well to make it work. Most people don't boot from the LSI controller because not having the boot rom speeds up POST but if you flash the card with the boot rom firmware then you can boot directly from a ZFS pool

If you can find a way of booting from SSD or nvme media then this would be a better bet than having the OS, the virtual machines files, and your main data all on spinning disks which won't be great from a performance viewpoint. I think there was a option for a rear drive cage on the DL380, might be worth looking into.

 

[thorro]

I installed Proxmox v8 without any problems on DL360 G9, P440ar put in the HBA mode (only had to delete the SmartArray data from drives before they could be seen by the Proxmox installer). It also boots from ZFS with no issues.

Now it seems the OS has direct control over drives. I can see all details with smartctl -a /dev/sda for example (output follows). Is there anything else the P440ar does in the background to make it not viable to use as a "JBOD" controller or it could have issues with ZFS in the future?

smartctl -a /dev/sda
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.5.11-8-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              HUC156060CSS200
Revision:             A703
Compliance:           SPC-4
User Capacity:        600,127,266,816 bytes [600 GB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        15030 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca0592b59a8
Serial number:        omitted
Device type:          disk
Transport protocol:   SAS (SPL-4)
Local Time is:        Sat Mar 23 08:43:03 2024 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     39 C
Drive Trip Temperature:        85 C

Accumulated power on time, hours:minutes 1530:59
Manufactured in week 22 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  23
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  83
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 219003066777600

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0      19897       5849.268           0
write:         0        0         0         0      20687       1222.266           0
verify:        0        0         0         0        525         24.220           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -    1530                 - [-   -    -]
# 2  Background short  Completed                   -    1530                 - [-   -    -]
# 3  Background short  Completed                   -    1529                 - [-   -    -]
# 4  Background short  Completed                   -    1529                 - [-   -    -]
# 5  Background short  Completed                   -    1529                 - [-   -    -]

Long (extended) Self-test duration: 4839 seconds [80.7 minutes]

 

[RolandK]

P440ar in hba mode is a bad idea

 

[thorro]

Bad experience? Not stable?

 

[RolandK]

sorry, i was too fast and without coffee in the morning. i don't have experience with 440ar, but for sure i can tell p410/420 has issues with hba mode

 

[thorro]

Np, I have bad experience with those, too, eventually replaced with LSI as it wouldn't even boot, if I remember correctly.

So, I genuinely want some info how 440ar behaves with ZFS in a long run as it seems OS sees the drives correctly/directly. Will do some tests, simulate bad drive etc.