downsides of using Smart Array P408i-a SR Gen10 in HBA mode - ZFS

[wonkyponky]

Hi peeps,

I want to setup a new Proxmox machine and I am looking at buying some refurbished HP ProLiant DL380 Gen10 which sports a Smart Array P408i-a SR Gen10 RAID controller.

I plan to use ZFS as base file system and read that the said controller takes the role of a HBA for all non-RAID disks and presents them as single devices to the OS.

So - theoretically all seems fine but here in the forums I read that you still recommend buying some used H220 or similar card with LSI controllers in IT mode.

What is the reason for that? Just because it has not been tested thoroughly? Of course I'd like to go the convenient way and not have to worry about installing a new device .. maybe the P408i also has better performance!?

Having said that you can get a used HP H220 in IT Mode, cabling included for around 50€

Thanks for Tipps!

 

[LnxBil]

I plan to use ZFS as base file system and read that the said controller takes the role of a HBA for all non-RAID disks and presents them as single devices to the OS.

If that is the case without configuring a RAID0 per disk, then it should be fine. We have the same with Fujitsu controllers and it looks totally fine.

 

[wonkyponky]

Hi there,
it seems so, read here, for example

I just learned that the P408i also controls up to 8x NVME drives via a riser card that this exact server has. This is of course the most tempting idea as those U.2 drives cost roughly the same as Enterprise SATA drives (around 320€ for 3.84 TB) but should be hell of a lot faster, not?

 

[wonkyponky]

well .. to answer my own question:
after some more research it seems that those controllers actually run a hybrid RAID/HBA mode in which some parameters of the drives won't be submitted to the OS which is not ideal for ZFS.

Might be just due to the drivers which some say are not perfect for this purpose .. see here for ex.

So the new question would be - which controller could I install in this machine to run a proper HBA mode with all the drives I might want to use? I have 8x SATA/SAS and 8x NVME backplanes. The NVME connected via PCIe riser card ..

 

[fireon]

It's been a while, but are you now using the Smart Array with ZFS?

 

[cosmos]

Same controller here as well, and in need for the same information since I'm trying to find out whether I'll use ZFS or not on my DL360Gen 10 with Smart Array P408i-a SR Gen10 RAID controller. Would be grateful if you @wonkyponky could share some info on what you've actually used and how that went...

 

[fireon]

I would also like to feedback on this topic.
I have been running my HP ML350G10 with ZFS and the Smart Array Controller with ZFS very successfully for about 2 months now, even under heavy load. The controller is “at least here” a hybrid. If you don't build a raid, you can use the installed SSDs 1:1. (HPE Smart Array P408i-a SR Gen10).

 

[wonkyponky]

Hi there,
for me it turned out like that:
- decided to NOT use the 408i as HBA for ZFS but created a "classical" RAID1 with onboard tools as boot/system drive for the PVE host
- created a 3 drive ZFS volume with the NVMe drives within Proxmox and use it as fast VM storage. No controller needed, the riser just acts as a breakout box and the drives control themselves.

In my case this made sense and I don't have to worry about the controller anymore. In the end I am also one of the guys that go for 200% safety and don't want to mess around with unsupported hardware. To me it sounded like "IT Mode or forget it" ..

 

[cosmos]

Thanks for the information. I went ahead and used the SSD in a hardware mirror setup, whereas left the HDDs as they are. Installed PVE to use the mirrored SSD as a boot XFS drive and then created a ZFS ZRAID pool from the 3 HDDs. Seems to be fine for the time being, will monitor.

 

[fireon]

@wonkyponky thanks for your feedback. And if your issue is resolved, please mark your thread as "solved". You can do this by editing your first post and setting the flag.

 

[wonkyponky]

Thanks for the information. I went ahead and used the SSD in a hardware mirror setup, whereas left the HDDs as they are. Installed PVE to use the mirrored SSD as a boot XFS drive and then created a ZFS ZRAID pool from the 3 HDDs. Seems to be fine for the time being, will monitor.

Allright, I hope this will work out for you!
@fireon I think my first post is too old - the option to edit it is missing ..

 

[yamazaki93]

Want to chime in here and share my experiences with this as I also set up PVE in a HPE DL380 Gen10 with P408i-a and ZFS. It's been running for about a month now and it's been working well. Before install PVE I had to delete all "logical volumes" with BIOS tool. Forced it into "mixed" mode and all drives are recognized and pass through to the PVE. OS install and boot properly on ZFS RAID1 and the rest of the drives are all added to another ZFS pool as data.

I did encounter an interesting event: One day I did an apt update and rebooted the system. Upon booting, suddenly I was getting 10 emails saying various ZFS drives offline, 5 drives lights lit up solid amber, a total scare. But then after it boots, ZFS tank shows status normal, all drives present and "ONLINE", all SMART value good, even HPE ILO reports HDD status indicator OK. Then after a second reboot, all amber lights disappeared, no emails about ZFS drive UNAVAIL. If anyone have similar experience, I'd love to hear your thoughts, I had assumed that it's some kind of message that got thrown during the boot sequence that tricked up the controller or some flaky connection.

 

[cosmos]

Want to chime in here and share my experiences with this as I also set up PVE in a HPE DL380 Gen10 with P408i-a and ZFS. It's been running for about a month now and it's been working well. Before install PVE I had to delete all "logical volumes" with BIOS tool. Forced it into "mixed" mode and all drives are recognized and pass through to the PVE. OS install and boot properly on ZFS RAID1 and the rest of the drives are all added to another ZFS pool as data.

Is the server having the latest firmware for P408i-a?

 

[fireon]

Is the server having the latest firmware for P408i-a?

Yes, I also recently updated to the latest firmware.

 

[yamazaki93]

Is the server having the latest firmware for P408i-a?

The firmware I was using was 6.22.

 

[cosmos]

The firmware I was using was 6.22.

Not an expert here, but perhaps you'd like to update to the latest version, which seems to be 7.11.