Solved - new Fujitsu 9300-8I SAS 12G SATA 6G PCI-e Controller HBA IT mode - crashing VM

C

chrischambers

New Member
Nov 1, 2022
15
2
3
I just purchase the following card Fujitsu 9300-8I SAS 12G SATA 6G PCI-e Controller HBA IT mode + 2*SFF8643 SATA from ALIExpress and when I place connect it to my Proxmox VM, - Proxmox hangs and then crash.

so the steps I have taken are:
I have checked my BIOS and made sure that my IOMMU is set to automatic
I then follow the steps from here and follow the steps for Legacy as my VM Bios is set to SeaBIOS
as my CPU is AMD I did the following steps

Code: 
nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
update-grub

with the following results

Code: 
Generating grub
configuration file ...
Found linux image:
/boot/vmlinuz-6.8.4-2-pve
Found initrd image:
/boot/initrd.img-6.8.4-2-pve
Found linux image:
/boot/vmlinuz-6.5.13-5-pve
Found initrd image:
/boot/initrd.img-6.5.13-5-pve
Found linux image:
/boot/vmlinuz-5.15.149-1-pve
Found initrd image:
/boot/initrd.img-5.15.149-1-pve
Found linux image:
/boot/vmlinuz-5.15.143-1-pve
Found initrd image:
/boot/initrd.img-5.15.143-1-pve
Found linux image:
/boot/vmlinuz-5.15.30-2-pve
Found initrd image:
/boot/initrd.img-5.15.30-2-pve
Found memtest86+
64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu
entry for UEFI Firmware Settings ...

then I

Code: 
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I tried to update proxmox-boot-tool refresh but this get me this, as I think becouse I am using the Legacy code.

Code: 
root@pve:~#
proxmox-boot-tool refresh
Running hook script
'proxmox-auto-removal'..
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates.
root@pve:~#
update-initramfs -u -k all
update-initramfs:
Generating /boot/initrd.img-6.8.4-2-pve
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates.
update-initramfs:
Generating /boot/initrd.img-6.5.13-5-pve
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates.
update-initramfs:
Generating /boot/initrd.img-5.15.149-1-pve
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates.
update-initramfs:
Generating /boot/initrd.img-5.15.143-1-pve
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates.
update-initramfs:
Generating /boot/initrd.img-5.15.30-2-pve
Running hook script
'zz-proxmox-boot'..
Re-executing
'/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount names                                                 
pace..
No
/etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
System booted in
EFI-mode but 'grub-efi-amd64' meta-package not installed!
Install
'grub-efi-amd64' to get updates. .

and so far so good no issues then I when to my VM (Unraid) and click on Add / PCI Devices / Raw Device

and located my LSI
1717775271663.png

I have tried both unticking and leaving the ROM-Bar Tick and each time the system stills hangs and crash.

I then restart my VM and then - my proxmox hangs. -
so at the moment as I am doing baby steps I have no hard drives hanging off the back of the LSI card.

I have a monitor plug into my proxmox server, and the errors I am getting are:
1717775649442.png


can anyone help ?










































































root@pve:~#[/CODE]























done[/CODE]
 
@santiagobiali thanks for your advice, but sadly it didn't work,

so here is what I did, I change my I440fx to q35 - but I was not sure about eh V-IOMMU option so I left this as default

1717831935497.png

1717832025873.png
rebooted and the proxmox and then added the SAS Controller - and started the VM, but sadly it rebooted the same issue. I then ran the blacklist mpt3sas command and repeated the above and again the same issue.

I then turned off my proxmox, and remove the SATA expansion card, and retry my unraid VM with the SAS controller and again the same error.

I then looked at the CPU Passthrough, and as this is all new to me, it looks OK so I will post the results here.

lspci

Code: 
root@pve:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 xHCI Compliant Host Controller (rev 01)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
20:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
20:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
21:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller (rev 11)
22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
25:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
26:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
26:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
27:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device
27:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 xHCI Compliant Host Controller
28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
28:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
28:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller

then to confirm that the modules are loaded

Code: 
root@pve:~#  lsmod | grep vfio
vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   69632  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                98304  1 vfio

then I rebooted and recheck
Code: 
root@pve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.076540] AMD-Vi: Unknown option - 'on'
[    0.192936] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
[    0.526367] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.532590] AMD-Vi: Extended features (0xf77ef22294ada, 0x0): PPR NX GT IA GA PC GA_vAPIC
[    0.532606] AMD-Vi: Interrupt remapping enabled
[    0.532754] AMD-Vi: Virtual APIC enabled
[    0.532861] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

to confirm remapping

Code: 
root@pve:~# dmesg | grep 'remapping'
[    0.532606] AMD-Vi: Interrupt remapping enabled

I look at this command lspci -nnk

Code: 
root@pve:~# lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) Root Complex [1462:7c02]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1462:7c02]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1462:7c02]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1462:7c02]
        Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
        Subsystem: Device [7c02:1462]
        Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
        Subsystem: Device [7c02:1462]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
        Subsystem: Micro-Star International Co., Ltd. [MSI] FCH SMBus Controller [1462:7c02]
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Micro-Star International Co., Ltd. [MSI] FCH LPC Bridge [1462:7c02]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 xHCI Compliant Host Controller [1022:43d5] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset USB 3.1 xHCI Compliant Host Controller [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset SATA Controller [1b21:1062]
        Kernel driver in use: ahci
        Kernel modules: ahci
03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Bridge [1b21:0201]
        Kernel driver in use: pcieport
20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
        Subsystem: ASMedia Technology Inc. 400 Series Chipset PCIe Port [1b21:3306]
        Kernel driver in use: pcieport
21:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
        DeviceName: Broadcom 5762
        Subsystem: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215]
        Kernel driver in use: ahci
        Kernel modules: ahci
22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: Micro-Star International Co., Ltd. [MSI] RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1462:7c02]
        Kernel driver in use: r8169
        Kernel modules: r8169
25:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097]
        Kernel modules: mpt3sas
26:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
        Subsystem: ASUSTeK Computer Inc. GK208B [GeForce GT 710] [1043:86cb]
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau
26:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
        Subsystem: ASUSTeK Computer Inc. GK208 HDMI/DP Audio Controller [1043:86cb]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Zeppelin/Raven/Raven2 PCIe Dummy Function [1462:7c02]
27:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device [1022:1456]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) Platform Security Processor (PSP) 3.0 Device [1462:7c02]
        Kernel driver in use: ccp
        Kernel modules: ccp
27:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 xHCI Compliant Host Controller [1022:145f]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Zeppelin USB 3.0 xHCI Compliant Host Controller [1462:7c02]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Zeppelin/Renoir PCIe Dummy Function [1462:7c02]
28:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
        Subsystem: Micro-Star International Co., Ltd. [MSI] FCH SATA Controller [AHCI mode] [1462:7c02]
        Kernel driver in use: ahci
        Kernel modules: ahci
28:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h (Models 00h-0fh) HD Audio Controller [1462:ec02]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

to me this should work, as I can see it in proxmox, but it will not pass through - what am I missing
 
Oh, that's the same kind of card I'm using in my desktop, though I'm not doing any kind of pass through with it.

Bash: 
# dmesg|grep LSISAS
[    2.141098] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.12.00), ChipRevision(0x02)

Have you checked what firmware version is on the card, and does it have both the BIOS and UEFI flashed to the card?

For reference, this should be the BIOS and UEFI flash images:

https://docs.broadcom.com/docs-and-...ckage_P16_IR_IT_FW_BIOS_for_MSDOS_Windows.zip

This is a very slightly newer firmware (info here) for the card over what's in the above .zip:

https://www.truenas.com/community/resources/lsi-9300-xx-firmware-update.145/download

And this is the most current release of the LSI/Broadcom storcli command line utility for doing stuff with the card, including for flashing firmware/bios/uefi images to it:

https://docs.broadcom.com/docs-and-.../sas_sata_nvme_12g_p31/STORCLI_SAS3.5_P31.zip

Actually went through that flashing process with the card yesterday, and it seems to be working fine. :)
 
Last edited:
@justinclift - thanks for replying I have tried that command within my proxmox terminal and it came back with nothing. I then type using sas or LSI and again nothing. in the end I did a dmesg and it came back with a load of information "see attachment for refer" which I have read through and I didn't see any reference to the LSI SAS card.

I am not sure on the current version of firmware I am on, but when the system is booting I do see the words IT flash across the screen when I see the LSI card.

before I set up update firmware I would like to get to see the current version, and if it is miles behind, then I will update, but thanks for the info and the links
 

Attachments

  • Dmesg.txt
    95.9 KB · Views: 2
Oh, there's no mention of mpt3sas in the dmesg output, even though it's showing up in the above lspci output:
chrischambers said:
25:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02) Subsystem: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097]
Kernel modules: mpt3sas
Click to expand...

I guess you blacklisted that mpt3sas module like @santiagobiali recommended above, then rebooted?

That'll stop the kernel on the host from loading the mpt3sas driver, which is useful when passing it through to a VM.

But it's the driver that shows the kernel version in the dmesg output (as the driver loads).

It's not the only way to get the firmware info though.

That storcli utility can do the job too:

Bash: 
# /opt/MegaRAID/storcli/storcli64 /c0 show | grep -i version
CLI Version = 007.3006.0000.0000 Apr 17, 2024
FW Version = 16.00.12.00
BIOS Version = 08.37.00.00_18.00.00.00
NVDATA Version = 14.01.00.07
Driver Version = 43.100.00.00

The /c0 there means "controller 0". ie the only SAS card in the system
 
As a data point, that dmesg output of yours indicates the cpu is a Ryzen 7 2700, the motherboard is an MSI B450 Tomahawk, and it's using an GeForce GT 710.

My desktop is a Ryzen 5950x with an Asrock B550M Pro4 and an Nvidia RTX 3070.

Not identical, but not tooooo far different either.

If it helps, my grub config line when I was mucking around with PCI passthrough a while ago was:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

I didn't actually have anything I needed passing through though, so not super sure if it'd actually work. :cool:
 
Last edited:
justinclift said:
Oh, there's no mention of mpt3sas in the dmesg output, even though it's showing up in the above lspci output:


I guess you blacklisted that mpt3sas module like @santiagobiali recommended above, then rebooted?

That'll stop the kernel on the host from loading the mpt3sas driver, which is useful when passing it through to a VM.

But it's the driver that shows the kernel version in the dmesg output (as the driver loads).

It's not the only way to get the firmware info though.

That storcli utility can do the job too:

Bash: 
# /opt/MegaRAID/storcli/storcli64 /c0 show | grep -i version
CLI Version = 007.3006.0000.0000 Apr 17, 2024
FW Version = 16.00.12.00
BIOS Version = 08.37.00.00_18.00.00.00
NVDATA Version = 14.01.00.07
Driver Version = 43.100.00.00

The /c0 there means "controller 0". ie the only SAS card in the system
Click to expand...

Yes I did following the installation which @santiagobiali mention and then rebooted. if this is wrong then how can we reverse this and get the mpt3sas working so that we can pass the card into the VM.

I also just tried the following command

Code: 
/opt/MegaRAID/storcli/storcli64 /c0 show | grep -i version

and i was informed that "No such file or directory

looking at your reference card, it looks the same :) I updated my grub with the following

1717857178959.png

and then updated the grub

Code: 
root@pve:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.8.4-2-pve
Found initrd image: /boot/initrd.img-6.8.4-2-pve
Found linux image: /boot/vmlinuz-6.5.13-5-pve
Found initrd image: /boot/initrd.img-6.5.13-5-pve
Found linux image: /boot/vmlinuz-5.15.149-1-pve
Found initrd image: /boot/initrd.img-5.15.149-1-pve
Found linux image: /boot/vmlinuz-5.15.143-1-pve
Found initrd image: /boot/initrd.img-5.15.143-1-pve
Found linux image: /boot/vmlinuz-5.15.30-2-pve
Found initrd image: /boot/initrd.img-5.15.30-2-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Adding boot menu entry for UEFI Firmware Settings ...

then I rebooted

I then added the SAS 3008 to my unraid and ( Oh my god she fired up - fingers cross )

so far so good -

once the server was up and running and before I started the raid I openned a terminal and type in lspci -nnk and got the following

Code: 
00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel modules: intel_agp
00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02)
        Subsystem: Red Hat, Inc. Device [1af4:1100]
00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: ehci-pci
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
00:1c.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
        Subsystem: Red Hat, Inc. QEMU PCIe Root port [1b36:0000]
        Kernel driver in use: pcieport
00:1c.1 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
        Subsystem: Red Hat, Inc. QEMU PCIe Root port [1b36:0000]
        Kernel driver in use: pcieport
00:1c.2 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
        Subsystem: Red Hat, Inc. QEMU PCIe Root port [1b36:0000]
        Kernel driver in use: pcieport
00:1c.3 PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c]
        Subsystem: Red Hat, Inc. QEMU PCIe Root port [1b36:0000]
        Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: uhci_hcd
00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: ehci-pci
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02)
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
05:01.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:02.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:03.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
05:04.0 PCI bridge [0604]: Red Hat, Inc. QEMU PCI-PCI bridge [1b36:0001]
06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]
        Subsystem: Red Hat, Inc. Virtio memory balloon [1af4:0005]
        Kernel driver in use: virtio-pci
06:05.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio SCSI [1af4:1004]
        Subsystem: Red Hat, Inc. Virtio SCSI [1af4:0008]
        Kernel driver in use: virtio-pci
06:10.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
06:12.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
        Subsystem: Red Hat, Inc. Virtio network device [1af4:0001]
        Kernel driver in use: virtio-pci
07:1b.0 USB controller [0c03]: Red Hat, Inc. QEMU XHCI Host Controller [1b36:000d] (rev 01)
        Subsystem: Red Hat, Inc. QEMU XHCI Host Controller [1af4:1100]
        Kernel driver in use: xhci_hcd

and it looks like the SAS2008 has been passed through.

Code: 
Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas


@justinclift thanks for your help so far. I am going to leave the system running for until tomorrow with no hard discs hanging off the back of the, card tomorrow I will added a test hard disc to see if it seen and usable and if all goes well I will move the rest over.

I will keep you all updated.
 
  • Like
Reactions: justinclift
chrischambers said:
I also just tried the following command

Code: 
/opt/MegaRAID/storcli/storcli64 /c0 show | grep -i version
Click to expand...
Ahhh.

Sorry, I didn't realise you're not yet super familiar with using the command line. The /opt/MegaRAID/storcli/storcli64 command is only present if you've installed the official package for it (storcli) from LSI / Broadcom.

That package is in the .zip file here, and is under the "Ubuntu" directory inside that. You install the package with this command:

Bash: 
# dpkg -i storcli_007.3006.0000.0000_all.deb

... but as you're not super familiar with command line usage (yet) I'm not sure if it's a good idea or not.

Then again, it seems like the grub change you did may have gotten things working well enough anyway. :)
 
chrischambers said:
it looks like the SAS2008 has been passed through
Click to expand...

Oh. You might be able to run the dmesg command inside the VM instead then, and get usable info out of it. ie:

Bash: 
dmesg | grep -i mpt3

Note that the -i option in that command tells it to do case insensitive filtering, so both MPT3, mpt3, mPt3, (etc) will all match. Without the -i, then only the specific case you give it will match.
 
Last edited:
@justinclift I just ran the dmesg | grep -i mpt3 within my VM Unraid and got this

Code: 
root@Tower:~# dmesg | grep -i mpt3
[   25.426428] mpt3sas version 43.100.00.00 loaded
[   25.429608] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (8152140 kB)
[   25.430118] mpt3sas_cm0 fault info from func: _base_wait_for_iocstate
[   25.430480] mpt3sas_cm0: fault_state(0x2667)!
[   25.430760] mpt3sas_cm0: sending diag reset !!
[   26.437648] mpt3sas_cm0: diag reset: SUCCESS
[   26.503387] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[   26.503713] mpt3sas_cm0: MSI-X vectors supported: 96
[   26.504140] mpt3sas_cm0:  0 4 4
[   26.511162] mpt3sas_cm0: High IOPs queues : disabled
[   26.511435] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 44
[   26.511682] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 45
[   26.511936] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 46
[   26.512179] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 47
[   26.512430] mpt3sas_cm0: iomem(0x00000000fdf80000), mapped(0x0000000020867eec), size(65536)
[   26.512752] mpt3sas_cm0: ioport(0x0000000000004000), size(256)
[   26.571634] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[   26.599913] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[   26.600461] mpt3sas_cm0: request pool(0x00000000d58ba81a) - dma(0x104400000): depth(10261), frame_size(128), pool_size(1282 kB)
[   26.645492] mpt3sas_cm0: sense pool(0x000000006930a2d0) - dma(0x117c00000): depth(10000), element_size(96), pool_size (937 kB)
[   26.646075] mpt3sas_cm0: reply pool(0x00000000962bca38) - dma(0x104600000): depth(10325), frame_size(128), pool_size(1290 kB)
[   26.646511] mpt3sas_cm0: config page(0x0000000092ca18dc) - dma(0x10631a000): size(512)
[   26.646784] mpt3sas_cm0: Allocated physical memory: size(27945 kB)
[   26.647025] mpt3sas_cm0: Current Controller Queue Depth(9997),Max Controller Queue Depth(10240)
[   26.647398] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[   26.812793] mpt3sas_cm0: _base_display_fwpkg_version: complete
[   26.813163] mpt3sas_cm0: overriding NVDATA EEDPTagMode setting
[   26.813959] mpt3sas_cm0: LSISAS3008: FWVersion(13.00.00.00), ChipRevision(0x02), BiosVersion(08.31.00.00)
[   26.814409] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[   26.817230] mpt3sas_cm0: sending port enable !!
[   29.367619] mpt3sas_cm0: hba_port entry: 00000000b21f4482, port: 255 is added to hba_port list
[   29.369627] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x5003005702807f00), phys(8)
[   34.497342] mpt3sas_cm0: port enable: SUCCESS

to me this is working ? and all I have to do now is power down the system , and then connect my hard discs ?
 
chrischambers said:
[ 26.813959] mpt3sas_cm0: LSISAS3008: FWVersion(13.00.00.00), ChipRevision(0x02), BiosVersion(08.31.00.00)
Click to expand...

Yep, that section (all the mpt3sas entries) means your VM is now using the HBA card correctly. So yep, you should be good to connect the hard drives and the disks should show up. :)

The specific line I've highlighted there has the version of the firmware on your card (13.00.00.00). That's exactly the same version as came on my card.

I didn't have any particular problems with it, so you'll probably be fine with that version too.

If the card does exhibit weird behaviour down the track, then we can look at upgrading the firmware on it. For now though, try adding your disks and get that all working. :D
 
Last edited:
@justinclift - many thanks for holding my hand on this. and walking me throw the steps.

if anyone is following this. my last steps were, that I recorded my hard drivers locations, and the reference SCSI passthrough ( the old way I did it ) on a bit of paper, then I powered down my server and removed the SATA cables and replace them with the breakout cables, again recording which HD is plug into which cable and socket.

next I had my fingers cross and powered up proxmox, NOTE: you will no longer see the LSI Hard drivers = then I powered up unraid.
once unraid was fired up, I when through and then assigned the right hard drive to the right hard drive slot.

then I hand to reset the Config, which clears the error message about "Too many wrong and/or missing disks!" and then start the array. This will then recreate the Parity drive.
 
  • Like
Reactions: justinclift
Awesome. Sounds like you got it all working in the end. :D

If that's indeed the case, would you be ok to update the thread title to [SOLVED] so future people can potentially better see it might help them out for their situation?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back