HPE MR416i-o Gen11 - FW FAULT crash (Fault code 0x10000) reproducible on kernels 6.14, 6.17 and 7.0

[mcalabrese]

Hello everyone.
We are experiencing a crash and reset of the megaraid_sas driver on a new HPE ProLiant DL380 Gen12 server after a certain amount of I/O. The controller enters a FAULT state with fault code 0x10000, resets, then recovers (but the crash repeats under load again).
We can reproduce the crash runing CrystalDiskMark inside a Windows 10 test VM, it usually takes about 5 test (9 passes, 4 GB size). Also reproducible with fio directly on the host using an SQL like workload (mix read and write).

What we have tried:

• Installed every firmware update available to date (controller, drives, BIOS, iLO)
• Tested kernels 6.14.11-9, 6.17.13-11 and 7.0.x (crash reproduces on all)
• Reinstalled Proxmox VE from scratch
• Purchased a communiy licence, activated the enterprise repo and installed every update
• Booted Ubuntu Server 26.04 LTS from ISO and could not reproduce the crash with the same fio workloads

According to iLO, controller and disk health are OK, but we are going to contact the seller and/or HPE support.

Some information:
- Current Kernel: Linux pve02 7.0.6-2-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.6-2 (2026-05-29T11:08Z) x86_64 GNU/Linux
- Storage: configured as LVM thin.
pveversion -v:

proxmox-ve: 9.2.0 (running kernel: 7.0.6-2-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0: 7.0.6-2
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-7-pve-signed: 7.0.2-7
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.14: 6.14.11-9
proxmox-kernel-6.14.11-9-pve-signed: 6.14.11-9
ceph-fuse: 19.2.3-pve4
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.13
libpve-guest-common-perl: 6.0.3
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.5
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-1
proxmox-backup-client: 4.2.1-1
proxmox-backup-file-restore: 4.2.1-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.3
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-4
pve-ha-manager: 5.2.4
pve-i18n: 3.7.5
pve-qemu-kvm: 11.0.0-3
pve-xtermjs: 6.0.0-1
qemu-server: 9.1.16
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.2-pve1

Test VM config:

agent: 1
bios: ovmf
boot: order=scsi0;sata0;sata1
cores: 2
cpu: x86-64-v2-AES
efidisk0: storage-lvm:vm-106-disk-3,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-10.1
memory: 8192
meta: creation-qemu=7.1.0,ctime=1670939025
name: w10-test2
net0: virtio=BC:24:119:967,bridge=vmbr0,tag=104
numa: 0
ostype: win11
sata0: none,media=cdrom
sata1: none,media=cdrom
scsi0: storage-lvm:vm-106-disk-4,discard=on,iothread=1,size=60G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=4ef93017-5b5c-4347-9363-b5841fc29fa9
sockets: 2
tpmstate0: storage-lvm:vm-106-disk-5,size=4M,version=v2.0
vmgenid: a484f0f3-21b6-4ed5-8381-b1d45ff7a3fc

Thank you!