Dual LSI 9300-8i IT mode HBAs with SAS3 Expander Disks not found in Proxmox v6.3

[SamCarson]

New install on Supermicro server completes normally. SAS drives show in Linux as 4 separate drives whose SCSI ids are identical. Multipath is installed. Linux does not set up multipath access, Therefore Proxmox cannot use the drives.

Proxmox v6.3-3
LSI 9300-8i IT firmware (v16)
Supermicro SAS3 Dual Expander Backplane

 

[SamCarson]

Solved. Disks formatted with 520 byte blocks. Reformatted with 512 byte blocks.

 

[auranext]

Hi SamCarson,

We have 2 cluster2nodes one on PVE5.3 the other on PVE6.2 running on HPE hardware (disk controller in HBA mode + 8 HPE enterprise class SSD) We encounter many instabilities.
sometimes an SSD is marked FAULTED by zfs, the CRC counter increments in the SMARTCTL output.
sometimes an SSD is abruptly ejected by the hpsa driver making the kernel unstable, the node can even reboot (via softdog)
log : PHYSICAL RESETTING ...
Systematically HP concludes that the SSD or the controller are fine
It seems to be a bug in HPSA or controller fw...
I am looking desperately for a reliable hardware for proxmox.
principally an HBA controller (zfs required) and SAS disks.
Can you share your experience on the hardware you encountered?

 

[SamCarson]

I setup Linux multiport disks (Supermicro SAS3 dual expander backplane & 2 HBA controllers). This hardware configuration yields 4 paths to each SAS drive. Proxmox does not recognize multiport disks. Using the Linux zfs command I set up the desired configuration outside of Proxmox. I then added the zfs storage to Proxmox. Seems to work ok. Still learning!
I don't know much about HP hardware. i have had the best luck with Supermicro server hardware.

The only enterprise SSDs in my system are NVME. They have their own controller (Supermicro AOC-SLG3-2M2 PCIe card). These appear to work ok with Proxmox.

Still testing.

 

[auranext]

thank you for this feedback
how long has your proxmox been in production?

did you check if the disks can be hot-swapped ?

 

[SamCarson]

Proxmox not in production yet.
Disks can be hat swapped.