PCIe passtrough problems - MSIX PBA outside of specified BAR

V

vanarebane

New Member
Jan 29, 2023
8
1
3
Hi, so I have spend whole day trying to find a solution.

Running Proxmox Virtual Environment 7.3-3

The setup is a Supermicro AOC-S2308L-L8i (9207-8I) RAID that shows up as LSI MegaRAID SAS 2108 - attached over a Thunderbolt 3 dock JHL7540.

There is a error when starting a VM, where I have passed though the RAID.

kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:07:00.0: hardware reports invalid configuration, MSIX PBA outside of specified BAR

Generally, it seems there is a problem with setting BAR memory size. There is also some overlap error

dmesg | grep -iC 3 "07:00.0"
Code: 
[    0.394737] pci 0000:06:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.394952] pci 0000:05:00.0: PCI bridge to [bus 06-09]
[    0.394964] pci 0000:05:00.0:   bridge window [mem 0xc4000000-0xc40fffff]
[    0.395062] pci 0000:07:00.0: [1000:0087] type 00 class 0x010700
[    0.395090] pci 0000:07:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.395111] pci 0000:07:00.0: reg 0x14: [mem 0x00000000-0x0000ffff 64bit]
[    0.395133] pci 0000:07:00.0: reg 0x1c: [mem 0x00000000-0x0003ffff 64bit]
[    0.395156] pci 0000:07:00.0: reg 0x30: [mem 0x00000000-0x000fffff pref]
[    0.395317] pci 0000:07:00.0: supports D1 D2
[    0.395432] pci 0000:07:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:03.1 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
[    0.395517] pci 0000:06:01.0: PCI bridge to [bus 07]
[    0.395633] pci 0000:08:00.0: [8086:15f0] type 00 class 0x0c0330
[    0.395665] pci 0000:08:00.0: reg 0x10: [mem 0xc4000000-0xc400ffff]
--
[    0.418969] AppArmor: AppArmor Filesystem Enabled
[    0.418997] pnp: PnP ACPI init
[    0.419100] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved
[    0.421312] pnp 00:02: disabling [io  0x0010-0x001f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421314] pnp 00:02: disabling [io  0x0022-0x003f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421315] pnp 00:02: disabling [io  0x0063] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421316] pnp 00:02: disabling [io  0x0065] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421317] pnp 00:02: disabling [io  0x0067-0x006f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421318] pnp 00:02: disabling [io  0x0072-0x007f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421319] pnp 00:02: disabling [io  0x0080] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421319] pnp 00:02: disabling [io  0x0084-0x0086] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421320] pnp 00:02: disabling [io  0x0088] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421321] pnp 00:02: disabling [io  0x008c-0x008e] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421322] pnp 00:02: disabling [io  0x0090-0x009f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421323] pnp 00:02: disabling [io  0x00a2-0x00bf] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421323] pnp 00:02: disabling [io  0x00b1] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421324] pnp 00:02: disabling [io  0x00e0-0x00ef] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421345] system 00:02: [io  0x04d0-0x04d1] has been reserved
[    0.421346] system 00:02: [io  0x040b] has been reserved
[    0.421347] system 00:02: [io  0x04d6] has been reserved
--
[    0.429821] pci 0000:06:01.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.429821] pci 0000:06:04.0: BAR 14: no space for [mem size 0x00200000]
[    0.429822] pci 0000:06:04.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.429823] pci 0000:07:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[    0.429824] pci 0000:07:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[    0.429825] pci 0000:07:00.0: BAR 3: no space for [mem size 0x00040000 64bit]
[    0.429825] pci 0000:07:00.0: BAR 3: failed to assign [mem size 0x00040000 64bit]
[    0.429826] pci 0000:07:00.0: BAR 1: no space for [mem size 0x00010000 64bit]
[    0.429827] pci 0000:07:00.0: BAR 1: failed to assign [mem size 0x00010000 64bit]
[    0.429828] pci 0000:07:00.0: BAR 0: assigned [io  0xb000-0xb0ff]
[    0.429833] pci 0000:06:01.0: PCI bridge to [bus 07]
[    0.429836] pci 0000:06:01.0:   bridge window [io  0xb000-0xbfff]
[    0.429845] pci 0000:06:01.0:   bridge window [mem 0x7fb0000000-0x7fb01fffff 64bit pref]
--
[    0.430944] pci 0000:06:01.0: Adding to iommu group 6
[    0.430965] pci 0000:06:02.0: Adding to iommu group 6
[    0.430987] pci 0000:06:04.0: Adding to iommu group 6
[    0.431026] pci 0000:07:00.0: Adding to iommu group 6
[    0.431064] pci 0000:08:00.0: Adding to iommu group 6
[    0.431089] pci 0000:65:00.0: Adding to iommu group 18
[    0.431098] pci 0000:65:00.1: Adding to iommu group 19
--
[   87.003422] fwbr103i0: port 2(tap103i0) entered disabled state
[   87.003476] fwbr103i0: port 2(tap103i0) entered blocking state
[   87.003477] fwbr103i0: port 2(tap103i0) entered forwarding state
[   87.571868] vfio-pci 0000:07:00.0: enabling device (0000 -> 0001)
[   87.679810] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x19@0x1e0
[   87.814619] fwbr103i0: port 2(tap103i0) entered disabled state
[   87.849916] fwbr103i0: port 1(fwln103i0) entered disabled state
[   87.849947] vmbr0: port 5(fwpr103p0) entered disabled state
--
[  206.357054] pcieport 0000:00:01.3:    [ 7] BadDLLP              
[  499.819389] r8169 0000:01:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
[  499.835760] r8169 0000:02:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
[  499.838390] vfio-pci 0000:07:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM


lspci -nv
Code: 
07:00.0 0107: 1000:0087 (rev 05)
        Subsystem: 1000:3020
        Physical Slot: 1
        Flags: fast devsel, IRQ 48, IOMMU group 6
        I/O ports at b000 [size=256]
        Memory at <unassigned> (64-bit, non-prefetchable) [disabled]
        Memory at <unassigned> (64-bit, non-prefetchable) [disabled]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable- Count=16 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [1e0] Secondary PCI Express
        Capabilities: [1c0] Power Budgeting <?>
        Capabilities: [190] Dynamic Power Allocation <?>
        Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas

lspci -nn
Code: 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b5] (rev 01)
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b6]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14cd]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14cd]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1679]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167a]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167b]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167c]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167d]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167e]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167f]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1680]
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO [144d:a80a]
04:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:0608]
05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
07:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
08:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018] [8086:15f0] (rev 06)
65:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1681] (rev d4)
65:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1640]
65:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
65:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161d]
65:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161e]
65:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 60)
65:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
65:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/Renoir Sensor Fusion Hub [1022:15e4]
66:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev a1)
67:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161f]
67:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d6]
67:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d7]
67:00.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:162e]
67:00.6 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:162f]




Things I have tried:

1. I also tried blocking the driver but same result (looks like the drivers are still there)

etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=1000:0087

etc/modprobe.d/vfio.conf
Code: 
softdep mpt3sas pre: vfio-pci

2. Following this tutorial, IMMOU and remapping is enabled

3. I checked, the RAID card is not alone in the IOMMU group 6. Everything connected to the Thunderbolt dock is in group 6.
 
I swapped out the OS disk with some other Linux to test if it was software or hardware. With other OS, the PCIe errors were the same - some IO overlap and failed BAR assignments.

The device is a Morefine M600 Mini PC.

In BIOS, there are some BAR Bus settings, that I don't understand well (and I have kept them what the default was). I have never seen so unrefined BIOS.
1706513082444.png

I get the Thunderbolt dock to turn on only when UCSI support is Enabled
1706513097583.png

Also I had to widget with PCIe Tunneling D3 and USB4 pre-CM settings
1706513129927.png
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom