Unable to boot installation environment Proxmox VE 8.0.2

[omcom]

Hello, I am new to Proxmox. I want to install 8.0.2 on Supermicro whitebox server with Asrock motherboard, Ryzen 5 CPU and Supermicro NVMe RAID card AOC-SLG3-2SM2. When I try to install from ISO, it stuck on some error related to megaraid_sas driver. I am not sure what's wrong. Windows PE and Debian 12 runs without problem. Is any way how to resolve it? Thank you.

 

[t.lamprecht]

Hmm odd that the others work, this message comes from a place where only the state reported by the device and it's firmware is checked, i.e., the kernel didn't really start to do things with that card.

Can you check if there's any firmware update for that device? You might need to contact the (system) vendor for that.

 

[omcom]

I've just flashed the HBA with latest firmware but nothing changed. It's interesting that Debian 12 shows similar error while installing but installation continues however virtual drive is not visible. Windows PE from Hierenn's boot CD boots normally without errors and Windows disk management is able to see Virtual drive. Any suggestions?

 

[t.lamprecht]

FWIW, I found some reports that fastboot (for uefi/bios firmware and/or windows) might leave the card in an odd state, as it does not get fully reset then, maybe try to turn that off.

https://forums.debian.net/viewtopic.php?p=780226&sid=2471e866e088b29424890a3cc8044d34#p780226
https://www.spinics.net/lists/linux-scsi/msg76204.html
https://docs.broadcom.com/doc/12353347#G7.159124 (the note in "3.1 Quick Configuration Steps for the BIOS Configuration Utility")

 

[omcom]

Disabling fast boot is good idea but I can't find it in my ASRock rack B650D4U BIOS config.

I found something about IOMMU config, so I had enabled it in BIOS but I don't know if it necesary to pass some kernel options to make it working.

I finally installed PVE 8, I had to wait for some megaraid_sas driver timeouts and after few minutes installation began. But I am not able to see virtual drive nor the HBA itself in storcli. It shows 0 controllers. Megaraid_sas module is installed.

Dnesg output:

[    1.200017] megasas: 07.719.03.00-rc1
[    1.200623] megaraid_sas 0000:01:00.0: BAR:0x0  BAR's base_addr(phys):0x00000000fce00000  mapped virt_addr:0x00000000d06aab28
[    1.200856] megaraid_sas 0000:01:00.0: FW now in Ready state
[    1.200859] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
[    1.201188] megaraid_sas 0000:01:00.0: firmware supports msix        : (128)
[    1.206466] megaraid_sas 0000:01:00.0: requested/available msix 20/20 poll_queue 0
[    1.206489]  megasas_alloc_irq_vectors+0x215/0x220 [megaraid_sas]
[    1.206497]  megasas_probe_one+0x851/0x1db0 [megaraid_sas]
[    1.206523]  megasas_init+0xd9/0xff0 [megaraid_sas]
[    1.206528]  ? __pfx_init_module+0x10/0x10 [megaraid_sas]
[    1.206578] megaraid_sas 0000:01:00.0: current msix/online cpus      : (20/12)
[    1.206579] megaraid_sas 0000:01:00.0: RDPQ mode     : (enabled)
[    1.206803] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 1517       LDIO threshold: 0
[    1.211506] megaraid_sas 0000:01:00.0: Performance mode :Balanced (latency index = 8)
[    1.211508] megaraid_sas 0000:01:00.0: FW supports sync cache        : Yes
[    1.211510] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[    1.212183] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa18450c0 flags=0x0020]
[    1.215761] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1844840 flags=0x0020]
[    1.216089] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa18378c0 flags=0x0020]
[    1.216376] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1840ac0 flags=0x0020]
[    1.217319] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa18431c0 flags=0x0020]
[    1.217323] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1842b00 flags=0x0020]
[    1.290094] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1836000 flags=0x0020]
[    1.290104] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1836300 flags=0x0020]
[    1.290108] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1836340 flags=0x0020]
[    1.290111] megaraid_sas 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0xa1836380 flags=0x0020]
[    1.384886] megaraid_sas 0000:01:00.0: FW supports atomic descriptor : Yes
[    8.420917] megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_get_ctrl_info 5385
[    8.420926] megaraid_sas 0000:01:00.0: Could not get controller info. Fail from megasas_init_adapter_fusion 1904
[    8.425264] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6542

 

[t.lamprecht]

There are indeed some report that this might fail with IOMMU, maybe try to pass the intel_iommu=off or amd_iommu=off kernel command line parameter, depending on your CPU.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992304#63

 

[omcom]

Thank you very much. The raid card and VD are finally visible and working in PVE after disabling IOMMU directly in BIOS. Should I be afraid about some affect in virtualization?

 

[t.lamprecht]

Thank you very much. The raid card and VD are finally visible and working in PVE after disabling IOMMU directly in BIOS. Should I be afraid about some affect in virtualization?

Not about virtualization directly, but PCI(e) passthrough is affected by such changes. If you do not care about that, you're fine.

FWICT, the hardware just really doesn't support running in systems with IOMMU, and I'm really not sure if there's anything that can be done in the kernel to avoid this (if anybody reading this found something, please holler at me, if it's somewhat sane we can test and apply it). In the end the vendor would need to step up here and provide a FW upgrade that fixes this, because IOMMU support is really becoming a commodity nowadays.