Dell PERC H755 firmware fatal error and controller reset after single SSD failure — Proxmox/Ceph node-wide I/O hang

[zabbix5674]

Hi everyone,





We had a strange storage issue on October 13 with a Dell R750 running Proxmox VE 9.2.21 and Ceph 20.2.4. The server uses a PERC H755 Front (firmware 52.21.1-5149) for both its OS RAID1 and Ceph SSDs. One SATA SSD used by OSD.4 started timing out and was marked offline. Shortly after, the PERC logged a fatal firmware error (SnapDumpHelper.c:360) and reset. Linux showed "FW in FAULT state" (0x10000), 166 outstanding I/Os, and system processes blocked for over 122 seconds. Node1's Ceph monitor temporarily lost quorum and Proxmox services became unresponsive. I expected a single failed OSD to be isolated, not affect the entire host. Has anyone seen this on an H755? Is it a known firmware/recovery issue, and is there a version that fixes it? Would moving Ceph disks to a dedicated IT-mode HBA and keeping OS storage on a separate controller be the better approach? I've attached verified, sanitized kernel, Ceph and PERC log excerpts. Any similar experiences or pointers would be appreciated.

 

[waltar]

dedicated ctrl's for raid or non-raid if use both same time usage is recommended way for

 

[jdancer]

Hi everyone,





We had a strange storage issue on October 13 with a Dell R750 running Proxmox VE 9.2.21 and Ceph 20.2.4. The server uses a PERC H755 Front (firmware 52.21.1-5149) for both its OS RAID1 and Ceph SSDs. One SATA SSD used by OSD.4 started timing out and was marked offline. Shortly after, the PERC logged a fatal firmware error (SnapDumpHelper.c:360) and reset. Linux showed "FW in FAULT state" (0x10000), 166 outstanding I/Os, and system processes blocked for over 122 seconds. Node1's Ceph monitor temporarily lost quorum and Proxmox services became unresponsive. I expected a single failed OSD to be isolated, not affect the entire host. Has anyone seen this on an H755? Is it a known firmware/recovery issue, and is there a version that fixes it? Would moving Ceph disks to a dedicated IT-mode HBA and keeping OS storage on a separate controller be the better approach? I've attached verified, sanitized kernel, Ceph and PERC log excerpts. Any similar experiences or pointers would be appreciated.

Always considered best practice to have OS and data/VMs on separate drives. Can use same controller. For example, I use two small drives to ZFS RAID-1 Promxox and rest of drives for OSDs.

As for the controllers, the drama with Dell PERC RAID controllers in HBA/IT-mode using the megaraid_sas driver seems to never end with these one-off cases.

At work, I swapped out all the Dell PERCs for Dell HBAs. They use the much simpler mpt3sas driver. No issues.

In your situation, the HBA you want is the Dell HBA 355i Front for 15th-gen Dells. For 16th-gen Dells, it's the Dell HBA465i. For 13th- & 14th-gen Dells, it's the Dell HBA330.