[SOLVED] MegaRAID 9540-8i virtual drive not found

[Noobtellabrot]

I want to install Proxmox 8.1 on a new server. However, the RAID controller virtual drives are not found. I tried Windows 10 and VMware ESXi 8.0 and there is no problem. I also tried Debian 12, Rocky Linux 9 and Ubuntu 22.04.4 LTS, but the virtual drive was not found in any of these Linux distributions.

Hardware:
CPU: AMD Ryzen 7900
Mainboard: ASRockRack B650D4U-2L2T/BCM
RAM: 4 x 32 GB Kingston ECC RAM
M.2 NVME (OS): WD 256 GB
RAID: Broadcom MegaRAID 9450-8i with 2 x Intel NVMe 8 TB U.2 SSD

 

[rj45]

Please provide:

#storcli64 show all
#journalctl -b --no-pager --grep "mega"
Firmware of 9450 updated?

 

[Noobtellabrot]

The controller is on the latest version available: 5.280.01-3972
Same for the BIOS: 4.09

storcli64 show all:

CLI Version = 007.2807.0000.0000 Dec 22, 2023
Operating system = Linux 6.5.11-8-pve
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = pve01
Operating System  = Linux 6.5.11-8-pve

journalctl -b --no-pager --grep "mega":

root@pve01:~# journalctl -b --no-pager --grep "mega"
Mar 14 08:24:26 pve01 kernel: efi: EFI v2.8 by American Megatrends
Mar 14 08:24:26 pve01 kernel: megasas: 07.725.01.00-rc1
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: BAR:0x0  BAR's base_addr(phys):0x000000fcf0700000  mapped virt_addr:0x00000000fe32718b
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: FW now in Ready state
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: firmware supports msix        : (128)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: requested/available msix 32/32 poll_queue 0
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: current msix/online cpus        : (32/24)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: RDPQ mode        : (enabled)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 1517         LDIO threshold: 0
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: Performance mode :Balanced (latency index = 8)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: FW supports sync cache        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa161ac00 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1619100 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1620540 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa161e840 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa161a880 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1611d80 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1612080 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1611000 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa16112c0 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000d address=0xa1611340 flags=0x0020]
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: FW supports atomic descriptor        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: FW provided supportMaxExtLDs: 0        max_lds: 32
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: controller type        : iMR(0MB)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: Online Controller Reset(OCR)        : Enabled
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: Secure JBOD support        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: NVMe passthru support        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: FW provided TM TaskAbort/Reset timeout        : 6 secs/60 secs
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: JBOD sequence map support        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: PCI Lane Margining support        : Yes
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: NVME page size        : (4096)
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
Mar 14 08:24:26 pve01 kernel: megaraid_sas 0000:01:00.0: INIT adapter done
Mar 14 08:24:26 pve01 kernel: usb 1-9: Manufacturer: American Megatrends Inc.
Mar 14 08:24:26 pve01 kernel: usb 1-9.3: Manufacturer: American Megatrends Inc.
Mar 14 08:24:26 pve01 kernel: usb 1-9.4: Manufacturer: American Megatrends Inc.
Mar 14 08:24:26 pve01 kernel: input: American Megatrends Inc. Virtual Mouse Gadget as /devices/pci0000:00/0000:00:02.1/0000:02:00.0/0000:03:0c.0/0000:0b:00.0/usb1/1-9/1-9.4/1-9.4:1.0/0003:046B:FF10.0003/input/input6
Mar 14 08:24:26 pve01 kernel: hid-generic 0003:046B:FF10.0003: input,hidraw2: USB HID v1.01 Mouse [American Megatrends Inc. Virtual Mouse Gadget] on usb-0000:0b:00.0-9.4/input0
Mar 14 08:24:26 pve01 kernel: usb 1-9.6: Manufacturer: American Megatrends Inc.
Mar 14 08:24:26 pve01 kernel: input: American Megatrends Inc. Virtual Keyboard Gadget as /devices/pci0000:00/0000:00:02.1/0000:02:00.0/0000:03:0c.0/0000:0b:00.0/usb1/1-9/1-9.6/1-9.6:1.0/0003:046B:FF10.0004/input/input7
Mar 14 08:24:26 pve01 kernel: hid-generic 0003:046B:FF10.0004: input,hidraw3: USB HID v1.01 Keyboard [American Megatrends Inc. Virtual Keyboard Gadget] on usb-0000:0b:00.0-9.6/input0
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: DCMD(opcode: 0x3010100) is timed out, func:megasas_issue_blocked_cmd
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_ld_list_query 4940
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: failed to get LD list
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: megasas_init_fw: megasas_get_device_list failed
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
Mar 14 08:25:19 pve01 kernel: megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6539
Mar 14 08:25:19 pve01 systemd-logind[884]: Watching system buttons on /dev/input/event7 (American Megatrends Inc. Virtual Keyboard Gadget)

 

[rj45]

6539 is fail_ioremap: https://codebrowser.dev/linux/linux/drivers/scsi/megaraid/megaraid_sas_base.c.html
uefi enabled?
Try with "intel_iommu=on iommu=pt" during setup

 

[Noobtellabrot]

Thanks @rj45 for your help.

I just found a blog with the solution: https://rephlex.de/blog/2021/10/16/fix-for-uefi-hardware-raid-linux-megaraid_sas-io_page_fault/

The Solution:

# Edit the file /etc/default/grub
nano /etc/default/grub
# change the variable GRUB_CMDLINE_LINUX_DEFAULT from GRUB_CMDLINE_LINUX_DEFAULT="quiet" to
# For AMD CPU
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt"
# For INTEL CPU
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"


# Then Update Grub and reboot
update-grub
reboot

Can you tell me if this solution is update safe?

 

[rj45]

yes, but you should add this to the end of existing parameters, mine is:
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_aspm=off intel_idle.max_cstate=0 intel_pstate=disable processor.max_cstate=1 mitigations=off intel_iommu=on iommu=pt"

 

[Noobtellabrot]

Alright, thank you.