[SOLVED] proxmox 7 / linux 5.11.22 issue with LSI 2008 controllers?

Hi ... I would be happy to but in full disclosure I'm a weekend warrior. When you say passthrough disabled can I simply shut down the UnRaid VM and then run dmesg? Or did you want me to turn off IOMMU in the bios?
 
You'd have to remove the vfio options in /etc/modprobe.d/ to make sure proxmox tried to claim the device. And un-blacklist the mpt3sas module if you've blacklisted it.

You may have to also edit your grub boot line depending on how you've configured it. But your iommu conf can stay as is
 
Last edited:
The only file I have in the modprobe.d directory is the blacklist.conf. The only entry in that file is:

blacklist nvidiafb

Also ... I am using System-boot not GRUB. I have not added any additional boot parameters.
 
I'm unfortunately not familiar with how system-boot is configured. Did you recall having to change any options to enable passthrough for your system?

If you run dmesg | grep mpt3 right now, what does that return?
 
I did not change anything. This is an AMD board that was originally running UnRaid native. If I recall .... back then I had to enable virtualization and change a setting to get better IOMMU groupings but I didn't mess with it recently before installing ProxMox. Here is the output:

1.310832] mpt3sas version 36.100.00.00 loaded
[ 1.604111] Modules linked in: fjes(-) crc32_pclmul mpt3sas(+) xhci_pci raid_class xhci_pci_renesas gpio_amdpt ahci i2c_piix4 alx scsi_transport_sas mdio xhci_hcd libahci wmi gpio_generic
[ 1.615874] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 1.616471] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 1.617085] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 1.622817] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12013/_scsih_probe()!
 
  • Like
Reactions: crabbypup
I'm gonna let this one play out for the time-being. If nothing develops before next Friday I will give it another shot. I have to get ready to travel and need this box up and running while I'm gone. I don't want to roll the dice this late on a Sunday because everything is working right now. Passing through the card to an UnRaid VM serves my purpose but I would eventually like to see those drives in ProxMox when UnRaid is off.

Appreciate the help and I'll follow along until next weekend. Thanks!
 
  • Like
Reactions: crabbypup
Hi, new to the forums but i am having similar issues white Proxmox 7, tested OK in Windows, Ubuntu and Proxmox 6.4

HPE ML350 Gen9
LSI 9201-16e (From a Dell SC8000) IT mode flashed to P20 and no BIOS on the card
External Dell sc200 disk chassi
 
Bobofistus said:
Hi, new to the forums but i am having similar issues white Proxmox 7, tested OK in Windows, Ubuntu and Proxmox 6.4

HPE ML350 Gen9
LSI 9201-16e (From a Dell SC8000) IT mode flashed to P20 and no BIOS on the card
External Dell sc200 disk chassi
Click to expand...
Can you post your dmesg log segments -
Code: 
dmesg | grep mpt3
to verify it's the same issue?

Can you try booting with the option
Code: 
mpt3sas.max_queue_depth=8000
appended to your kernel boot line and see if the issue persists?

Lastly, have you tried the edge kernel?
 
  • Like
Reactions: Bobofistus
crabbypup said:
Can you post your dmesg log segments -
Code: 
dmesg | grep mpt3
to verify it's the same issue?
Click to expand...
dmesg | grep mpt
[ 0.024469] Device empty
[ 5.700316] mpt3sas version 36.100.00.00 loaded
[ 5.706268] mpt3sas 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.707432] mpt3sas 0000:04:00.0: enabling device (0140 -> 0142)
[ 5.707548] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.768419] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.768436] mpt2sas_cm0: MSI-X vectors supported: 1
[ 5.768442] mpt2sas_cm0: 0 1
[ 5.768584] mpt2sas_cm0: High IOPs queues : disabled
[ 5.768586] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.768589] mpt2sas_cm0: iomem(0x0000000092e1c000), mapped(0x000000007b5c699d), size(16384)
[ 5.768595] mpt2sas_cm0: ioport(0x0000000000002000), size(256)
[ 5.815525] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.843024] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 5.843075] Modules linked in: ses(+) enclosure crc32_pclmul mpt3sas(+) xhci_pci ahci hpsa raid_class xhci_pci_renesas uhci_hcd i2c_i801 ehci_pci i2c_smbus libahci lpc_ich tg3 xhci_hcd ehci_hcd be2net(+) scsi_transport_sas wmi
[ 5.843202] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 5.843242] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 5.843272] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 5.843551] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
[ 5.908263] mpt3sas 0000:81:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.908524] mpt3sas 0000:81:00.0: enabling device (0140 -> 0142)
[ 5.908659] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.963144] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.963158] mpt2sas_cm1: MSI-X vectors supported: 1
[ 5.963164] mpt2sas_cm1: 0 1
[ 5.963345] mpt2sas_cm1: High IOPs queues : disabled
[ 5.963348] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.963350] mpt2sas_cm1: iomem(0x00000000c821c000), mapped(0x000000004569a785), size(16384)
[ 5.963357] mpt2sas_cm1: ioport(0x000000000000a000), size(256)
[ 6.018886] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 6.046941] mpt2sas_cm1: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 6.047175] mpt2sas_cm1: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
root@xxx:~#
crabbypup said:
Can you try booting with the option
Code: 
mpt3sas.max_queue_depth=8000
appended to your kernel boot line and see if the issue persists?
Click to expand...
I am using UEFI so nano /etc/kernel/cmdline root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on mpt3sas.max_queue_depth=10000
If you are using BIOS add it in grub, nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet mpt3sas.max_queue_depth=10000"

It fixed the issue:D so works on my host and also on my test Proxmox-VM inside Proxmox (using HBA passthrough)

crabbypup said:
Lastly, have you tried the edge kernel?
Click to expand...
No, i can try on my test Proxmox if you give me a guide:D (Linux newbie)

i searched Google for quiet mpt3sas.max_queue_depth= and found some articles, seams like a known bug
https://www.reddit.com/r/homelab/comments/j1d0i6/fyi_the_linux_58_kernel_currently_doesnt_work/
https://unraid.net/blog/unraid-6-9-beta30
 
  • Like
Reactions: crabbypup
Bobofistus said:
dmesg | grep mpt
[ 0.024469] Device empty
[ 5.700316] mpt3sas version 36.100.00.00 loaded
[ 5.706268] mpt3sas 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.707432] mpt3sas 0000:04:00.0: enabling device (0140 -> 0142)
[ 5.707548] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.768419] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.768436] mpt2sas_cm0: MSI-X vectors supported: 1
[ 5.768442] mpt2sas_cm0: 0 1
[ 5.768584] mpt2sas_cm0: High IOPs queues : disabled
[ 5.768586] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.768589] mpt2sas_cm0: iomem(0x0000000092e1c000), mapped(0x000000007b5c699d), size(16384)
[ 5.768595] mpt2sas_cm0: ioport(0x0000000000002000), size(256)
[ 5.815525] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.843024] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 5.843075] Modules linked in: ses(+) enclosure crc32_pclmul mpt3sas(+) xhci_pci ahci hpsa raid_class xhci_pci_renesas uhci_hcd i2c_i801 ehci_pci i2c_smbus libahci lpc_ich tg3 xhci_hcd ehci_hcd be2net(+) scsi_transport_sas wmi
[ 5.843202] base_alloc_rdpq_dma_pool+0xef/0x18c [mpt3sas]
[ 5.843242] mpt3sas_base_attach.cold+0x3d9/0x15f5 [mpt3sas]
[ 5.843272] _scsih_probe+0x6f2/0x8d0 [mpt3sas]
[ 5.843551] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
[ 5.908263] mpt3sas 0000:81:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 5.908524] mpt3sas 0000:81:00.0: enabling device (0140 -> 0142)
[ 5.908659] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (197928948 kB)
[ 5.963144] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 5.963158] mpt2sas_cm1: MSI-X vectors supported: 1
[ 5.963164] mpt2sas_cm1: 0 1
[ 5.963345] mpt2sas_cm1: High IOPs queues : disabled
[ 5.963348] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 125
[ 5.963350] mpt2sas_cm1: iomem(0x00000000c821c000), mapped(0x000000004569a785), size(16384)
[ 5.963357] mpt2sas_cm1: ioport(0x000000000000a000), size(256)
[ 6.018886] mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[ 6.046941] mpt2sas_cm1: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 6.047175] mpt2sas_cm1: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12037/_scsih_probe()!
root@xxx:~#

I am using UEFI so nano /etc/kernel/cmdline root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on mpt3sas.max_queue_depth=10000
If you are using BIOS add it in grub, nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet mpt3sas.max_queue_depth=10000"

It fixed the issue:D so works on my host and also on my test Proxmox-VM inside Proxmox (using HBA passthrough)


No, i can try on my test Proxmox if you give me a guide:D (Linux newbie)

i searched Google for quiet mpt3sas.max_queue_depth= and found some articles, seams like a known bug
https://www.reddit.com/r/homelab/comments/j1d0i6/fyi_the_linux_58_kernel_currently_doesnt_work/
https://unraid.net/blog/unraid-6-9-beta30
Click to expand...
Excellent!

To summarize for anyone else, yes it's the exact same issue.

It appears to be specifically related to the use of external SAS enclosures with certain controllers, of which the 2008 is one. The edge kernel doesn't appear to be necessary to resolve the issue - it's enough to add that parameter to your boot line.

No need to test the edge kernel as long as what you have is working.
 
Last edited:
Better late then never .... Finally got around to fiddling with this. For those searching at a later date let me add:

If you are using an AMD CPU the correct entry in the cmdline file is:

root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on mpt3sas.max_queue_depth=10000

However ... I believe my issue was the lack of running the following after saving the file:

proxmox-boot-tool refresh

The GRUB portion as listed should be fine but make sure you run the following after saving the file:

update-grub
 
  • Like
Reactions: clipz98 and Docop2
Hello, I think I am having a similar issue. I tried adding a LSI00300 IT Mode LSI 9207-8E 6Gb/s Extelnal PCI-E 3.0x8 Host Controller Card to our Dell R720XD server running Proxmox VE 8 with TrueNAS to add a NetApp DS2446 with drives in it. This card ended up taking down our whole server. I tried redoing the Proxmox host over, then restoring backups, but the problem came back. I figured out it was this card that was causing the whole system to go down. I'm not sure if this card was not flashed for IT mode correctly, or if it is just not compatible with my system. Could anyone recommend an HBA Controller Card that does not have issues like this? Or a solution to make this card work.

Thank you,

2023-07-30_21-41-21.jpg2023-07-30_21-46-13.jpg2023-07-31_0-39-41.jpg2023-07-31_0-58-07.jpg
 
CryptoVibe said:
Hello, I think I am having a similar issue. I tried adding a LSI00300 IT Mode LSI 9207-8E 6Gb/s Extelnal PCI-E 3.0x8 Host Controller Card to our Dell R720XD server running Proxmox VE 8 with TrueNAS to add a NetApp DS2446 with drives in it. This card ended up taking down our whole server. I tried redoing the Proxmox host over, then restoring backups, but the problem came back. I figured out it was this card that was causing the whole system to go down. I'm not sure if this card was not flashed for IT mode correctly, or if it is just not compatible with my system. Could anyone recommend an HBA Controller Card that does not have issues like this? Or a solution to make this card work.

Thank you,

View attachment 53853View attachment 53854View attachment 53855View attachment 53856
Click to expand...
Have you tried the documented fix that's shared in this thread?
 
crabbypup said:
Have you tried the documented fix that's shared in this thread?
Click to expand...
I saw this, but did not try it yet. I just got my system back up today, after being down for almost a week. I think you're talking about adding this:

mpt3sas.max_queue_depth=10000

To this:
nano /etc/default/grub

So it looks like this:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on mpt3sas.max_queue_depth=10000"

I will try that tonight if that looks good to you. I am using the most up to date Proxmox / TrueNAS available. I"m not sure what kernal, I'm pretty new to Linux / Proxmox / TrueNAS / Ubuntu. I've just been learning it to farm Chia.

Thank you!
 
CryptoVibe said:
I think you're talking about adding this:

mpt3sas.max_queue_depth=10000

To this:
nano /etc/default/grub

So it looks like this:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on mpt3sas.max_queue_depth=10000"
Click to expand...
Yes, that's correct.

If you're doing pcie passthrough for the controller to truenas, you'll want to make sure the correct controller is being claimed by the vfio stub driver during boot, since the OS usually doesn't like it when the device it's booting off of disappears suddenly. That's out of scope for this thread, but it's documented on the proxmox wiki.
 
That makes complete sense. I have GPU's & other PCIe cards working with passthrough. I had this devices number / codes added to this:

echo "options vfio-pci ids=10de:24b0,10de:228b,10de:2503,10de:228e,10de:2489,10de:228b,10b5:8747,1bb1:5016 disable_vga=1"> /etc/modprobe.d/vfio.conf

But what you pointed out I did not try yet. I will try it tonight, thank you very much!
 
crabbypup said:
Yes, that's correct.

If you're doing pcie passthrough for the controller to truenas, you'll want to make sure the correct controller is being claimed by the vfio stub driver during boot, since the OS usually doesn't like it when the device it's booting off of disappears suddenly. That's out of scope for this thread, but it's documented on the proxmox wiki.
Click to expand...
I tried what you pointed out, but as soon as I did pass through & restarted TrueNAS, the whole server went down with the same errors. I took the Controller Card back out & the server started up / is running now again...

Does anyone know some specific IT Mode LSI 9207-8E cards that will work in this Dell R720XD server running Proxmox VE 8 with TrueNAS to add a NetApp DS2446?

Thank you,
 
You must log in or register to reply here.
Top Bottom