[SOLVED] proxmox 7 / linux 5.11.22 issue with LSI 2008 controllers?

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
Hi ... I would be happy to but in full disclosure I'm a weekend warrior. When you say passthrough disabled can I simply shut down the UnRaid VM and then run dmesg? Or did you want me to turn off IOMMU in the bios?
 

crabbypup

New Member
Aug 2, 2021
16
3
3
30
You'd have to remove the vfio options in /etc/modprobe.d/ to make sure proxmox tried to claim the device. And un-blacklist the mpt3sas module if you've blacklisted it.

You may have to also edit your grub boot line depending on how you've configured it. But your iommu conf can stay as is
 
Last edited:

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
The only file I have in the modprobe.d directory is the blacklist.conf. The only entry in that file is:

blacklist nvidiafb

Also ... I am using System-boot not GRUB. I have not added any additional boot parameters.
 

crabbypup

New Member
Aug 2, 2021
16
3
3
30
I'm unfortunately not familiar with how system-boot is configured. Did you recall having to change any options to enable passthrough for your system?

If you run dmesg | grep mpt3 right now, what does that return?
 

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
I did not change anything. This is an AMD board that was originally running UnRaid native. If I recall .... back then I had to enable virtualization and change a setting to get better IOMMU groupings but I didn't mess with it recently before installing ProxMox. Here is the output:

1.310832] mpt3sas version 36.100.00.00 loaded
[ 1.604111] Modules linked in: fjes(-) crc32_pclmul mpt3sas(+) xhci_pci raid_class xhci_pci_renesas gpio_amdpt ahci i2c_piix4 alx scsi_transport_sas mdio xhci_hcd libahci wmi gpio_generic
[ 1.615874] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 1.616471] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 1.617085] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 1.622817] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12013/_scsih_probe()!
 
  • Like
Reactions: crabbypup

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
Right now I'm back on Linux 5.11.22-3-pve but I did try the 5.13 last night.
 

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
I'm gonna let this one play out for the time-being. If nothing develops before next Friday I will give it another shot. I have to get ready to travel and need this box up and running while I'm gone. I don't want to roll the dice this late on a Sunday because everything is working right now. Passing through the card to an UnRaid VM serves my purpose but I would eventually like to see those drives in ProxMox when UnRaid is off.

Appreciate the help and I'll follow along until next weekend. Thanks!
 
  • Like
Reactions: crabbypup

Bobofistus

New Member
Nov 12, 2021
2
1
3
24
Hi, new to the forums but i am having similar issues white Proxmox 7, tested OK in Windows, Ubuntu and Proxmox 6.4

HPE ML350 Gen9
LSI 9201-16e (From a Dell SC8000) IT mode flashed to P20 and no BIOS on the card
External Dell sc200 disk chassi
 

crabbypup

New Member
Aug 2, 2021
16
3
3
30
Hi, new to the forums but i am having similar issues white Proxmox 7, tested OK in Windows, Ubuntu and Proxmox 6.4

HPE ML350 Gen9
LSI 9201-16e (From a Dell SC8000) IT mode flashed to P20 and no BIOS on the card
External Dell sc200 disk chassi
Can you post your dmesg log segments -
Code:
dmesg | grep mpt3
to verify it's the same issue?

Can you try booting with the option
Code:
mpt3sas.max_queue_depth=8000
appended to your kernel boot line and see if the issue persists?

Lastly, have you tried the edge kernel?
 
  • Like
Reactions: Bobofistus

Bobofistus

New Member
Nov 12, 2021
2
1
3
24
Can you post your dmesg log segments -
Code:
dmesg | grep mpt3
to verify it's the same issue?
dmesg | grep mpt
[ 0.024469] Device empty
[ 5.700316] mpt3sas version 36.100.00.00 loaded
[ 5.706268] mpt3sas 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.707432] mpt3sas 0000:04:00.0: enabling device (0140 -> 0142)
[ 5.707548] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.768419] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.768436] mpt2sas_cm0: MSI-X vectors supported: 1
[ 5.768442] mpt2sas_cm0: 0 1
[ 5.768584] mpt2sas_cm0: High IOPs queues : disabled
[ 5.768586] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.768589] mpt2sas_cm0: iomem(0x0000000092e1c000), mapped(0x000000007b5c699d), size(16384)
[ 5.768595] mpt2sas_cm0: ioport(0x0000000000002000), size(256)
[ 5.815525] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.843024] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 5.843075] Modules linked in: ses(+) enclosure crc32_pclmul mpt3sas(+) xhci_pci ahci hpsa raid_class xhci_pci_renesas uhci_hcd i2c_i801 ehci_pci i2c_smbus libahci lpc_ich tg3 xhci_hcd ehci_hcd be2net(+) scsi_transport_sas wmi
[ 5.843202] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 5.843242] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 5.843272] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 5.843551] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
[ 5.908263] mpt3sas 0000:81:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.908524] mpt3sas 0000:81:00.0: enabling device (0140 -> 0142)
[ 5.908659] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.963144] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.963158] mpt2sas_cm1: MSI-X vectors supported: 1
[ 5.963164] mpt2sas_cm1: 0 1
[ 5.963345] mpt2sas_cm1: High IOPs queues : disabled
[ 5.963348] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.963350] mpt2sas_cm1: iomem(0x00000000c821c000), mapped(0x000000004569a785), size(16384)
[ 5.963357] mpt2sas_cm1: ioport(0x000000000000a000), size(256)
[ 6.018886] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 6.046941] mpt2sas_cm1: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 6.047175] mpt2sas_cm1: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
root@xxx:~#
Can you try booting with the option
Code:
mpt3sas.max_queue_depth=8000
appended to your kernel boot line and see if the issue persists?
I am using UEFI so nano /etc/kernel/cmdline root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on mpt3sas.max_queue_depth=10000
If you are using BIOS add it in grub, nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet mpt3sas.max_queue_depth=10000"

It fixed the issue:D so works on my host and also on my test Proxmox-VM inside Proxmox (using HBA passthrough)

Lastly, have you tried the edge kernel?
No, i can try on my test Proxmox if you give me a guide:D (Linux newbie)

i searched Google for quiet mpt3sas.max_queue_depth= and found some articles, seams like a known bug
https://www.reddit.com/r/homelab/comments/j1d0i6/fyi_the_linux_58_kernel_currently_doesnt_work/
https://unraid.net/blog/unraid-6-9-beta30
 
  • Like
Reactions: crabbypup

crabbypup

New Member
Aug 2, 2021
16
3
3
30
dmesg | grep mpt
[ 0.024469] Device empty
[ 5.700316] mpt3sas version 36.100.00.00 loaded
[ 5.706268] mpt3sas 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.707432] mpt3sas 0000:04:00.0: enabling device (0140 -> 0142)
[ 5.707548] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.768419] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.768436] mpt2sas_cm0: MSI-X vectors supported: 1
[ 5.768442] mpt2sas_cm0: 0 1
[ 5.768584] mpt2sas_cm0: High IOPs queues : disabled
[ 5.768586] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.768589] mpt2sas_cm0: iomem(0x0000000092e1c000), mapped(0x000000007b5c699d), size(16384)
[ 5.768595] mpt2sas_cm0: ioport(0x0000000000002000), size(256)
[ 5.815525] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.843024] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 5.843075] Modules linked in: ses(+) enclosure crc32_pclmul mpt3sas(+) xhci_pci ahci hpsa raid_class xhci_pci_renesas uhci_hcd i2c_i801 ehci_pci i2c_smbus libahci lpc_ich tg3 xhci_hcd ehci_hcd be2net(+) scsi_transport_sas wmi
[ 5.843202] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 5.843242] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 5.843272] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 5.843551] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
[ 5.908263] mpt3sas 0000:81:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.908524] mpt3sas 0000:81:00.0: enabling device (0140 -> 0142)
[ 5.908659] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.963144] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.963158] mpt2sas_cm1: MSI-X vectors supported: 1
[ 5.963164] mpt2sas_cm1: 0 1
[ 5.963345] mpt2sas_cm1: High IOPs queues : disabled
[ 5.963348] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.963350] mpt2sas_cm1: iomem(0x00000000c821c000), mapped(0x000000004569a785), size(16384)
[ 5.963357] mpt2sas_cm1: ioport(0x000000000000a000), size(256)
[ 6.018886] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 6.046941] mpt2sas_cm1: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 6.047175] mpt2sas_cm1: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
root@xxx:~#

I am using UEFI so nano /etc/kernel/cmdline root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on mpt3sas.max_queue_depth=10000
If you are using BIOS add it in grub, nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet mpt3sas.max_queue_depth=10000"

It fixed the issue:D so works on my host and also on my test Proxmox-VM inside Proxmox (using HBA passthrough)


No, i can try on my test Proxmox if you give me a guide:D (Linux newbie)

i searched Google for quiet mpt3sas.max_queue_depth= and found some articles, seams like a known bug
https://www.reddit.com/r/homelab/comments/j1d0i6/fyi_the_linux_58_kernel_currently_doesnt_work/
https://unraid.net/blog/unraid-6-9-beta30
Excellent!

To summarize for anyone else, yes it's the exact same issue.

It appears to be specifically related to the use of external SAS enclosures with certain controllers, of which the 2008 is one. The edge kernel doesn't appear to be necessary to resolve the issue - it's enough to add that parameter to your boot line.

No need to test the edge kernel as long as what you have is working.
 
Last edited:

Cr4fty

New Member
Aug 22, 2021
7
4
3
58
Better late then never .... Finally got around to fiddling with this. For those searching at a later date let me add:

If you are using an AMD CPU the correct entry in the cmdline file is:

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on mpt3sas.max_queue_depth=10000

However ... I believe my issue was the lack of running the following after saving the file:

proxmox-boot-tool refresh

The GRUB portion as listed should be fine but make sure you run the following after saving the file:

update-grub
 
  • Like
Reactions: clipz98 and Docop2

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!