HBA Passthru Issues after upgrade to 6

[Random777]

Hey All,

I've hit a brick wall here and not to sure what else i can do.

I've updated all my nodes to Proxmox 6, After some issues i got most of thing running.

My main and big issue with Passthru of my HBA.

Before the upgrade:
I had a fileserver running 2012R2 with my HBA passed through to it, using Stablebit Drive Pool to host my file share. This was working great for me and i was fan of it.(there may be better solutions out there, i'm not looking to change, im looking to repair the issue.)

After the upgrade:
I can boot the file server VM, the HBA card is detected in windows but i can not see any hard drives. When i boot the VM i see the bios on the HBA and see the disk get detected (Like it did before), Just no disks in windows once it boots.

When the host first boots, I can see the disks listed under the Disks section in the webgui, when i boot the VM with the pcie passthru, the disk get removed from there.

I've tried enabling / disabling rombar with no luck.
I've looked thru the PCI passthru guide on the wiki and confirmed that stuff is still there.
I've reseated the HBA to a different pci slot and had it change positions with no change.
Atm there is nothing else running on the host.

pveversion

pve-manager/6.0-4/2a719255 (running kernel: 5.0.15-1-pve)

iommu group

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port [8086:3405] (rev 13)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 [8086:3408] (rev 13)
IOMMU Group 10:
        00:1f.0 ISA bridge [0601]: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller [8086:3a16]
        00:1f.2 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 [8086:3a20]
        00:1f.3 SMBus [0c05]: Intel Corporation 82801JI (ICH10 Family) SMBus Controller [8086:3a30]
        00:1f.5 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 [8086:3a26]
IOMMU Group 11:
        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
        02:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 12:
        03:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B [11ab:6485] (rev 01)
IOMMU Group 13:
        ff:00.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers [8086:2c41] (rev 05)
        ff:00.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder [8086:2c01] (rev 05)
IOMMU Group 14:
        ff:02.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 QPI Link 0 [8086:2c10] (rev 05)
        ff:02.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 [8086:2c11] (rev 05)
IOMMU Group 15:
        ff:03.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller [8086:2c18] (rev 05)
        ff:03.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder [8086:2c19] (rev 05)
        ff:03.4 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers [8086:2c1c] (rev 05)
IOMMU Group 16:
        ff:04.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers [8086:2c20] (rev 05)
        ff:04.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers [8086:2c21] (rev 05)
        ff:04.2 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers [8086:2c22] (rev 05)
        ff:04.3 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers [8086:2c23] (rev 05)
IOMMU Group 17:
        ff:05.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers [8086:2c28] (rev 05)
        ff:05.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers [8086:2c29] (rev 05)
        ff:05.2 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers [8086:2c2a] (rev 05)
        ff:05.3 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers [8086:2c2b] (rev 05)
IOMMU Group 18:
        ff:06.0 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers [8086:2c30] (rev 05)
        ff:06.1 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers [8086:2c31] (rev 05)
        ff:06.2 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers [8086:2c32] (rev 05)
        ff:06.3 Host bridge [0600]: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers [8086:2c33] (rev 05)
IOMMU Group 2:
        00:03.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 [8086:340a] (rev 13)
IOMMU Group 3:
        00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 13)
IOMMU Group 4:
        00:14.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] (rev 13)
        00:14.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] (rev 13)
        00:14.2 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] (rev 13)
        00:14.3 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers [8086:3438] (rev 13)
IOMMU Group 5:
        00:1a.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 [8086:3a37]
        00:1a.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 [8086:3a38]
        00:1a.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a39]
        00:1a.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 [8086:3a3c]
IOMMU Group 6:
        00:1b.0 Audio device [0403]: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller [8086:3a3e]
IOMMU Group 7:
        00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
        00:1c.2 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3 [8086:3a44]
        00:1c.3 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 4 [8086:3a46]
        00:1c.4 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 [8086:3a48]
        04:00.0 SATA controller [0106]: JMicron Technology Corp. JMB363 SATA/IDE Controller [197b:2363] (rev 03)
        04:00.1 IDE interface [0101]: JMicron Technology Corp. JMB363 SATA/IDE Controller [197b:2363] (rev 03)
        05:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6315 Series Firewire Controller [1106:3403]
        06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 02)
IOMMU Group 8:
        00:1d.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a34]
        00:1d.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a35]
        00:1d.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a36]
        00:1d.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 [8086:3a3a]
IOMMU Group 9:
        00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 90)

/etc/modprobe.d/vfio.conf

options vfio-pci ids=11ab:6485
options vfio_iommu_type1 allow_unsafe_interrupts=1

lspci -nn -s 3:00

03:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B [11ab:6485] (rev 01)

vm conf

agent: 1
bootdisk: scsi0
cores: 2
hostpci0: 03:00.0,pcie=1
machine: q35
memory: 4096
name: fs1
net0: virtio=E2:E9:88:AD:65:43,bridge=vmbr0
numa: 0
ostype: win8
scsi0: local-lvm:vm-100-disk-1,cache=writeback,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=43221c22-5551-4686-82a6-3fc76a26a913
sockets: 1
startup: order=1

Please let me know if you need anymore info specifically.

 

[Random777]

Hey all,

Just an update, a reddit user was able to solve the issue for me.


This is a backwards-incompatible change (bug) in the "q35" machine model in QEMU 4. To fix it you need to add this line to your VM's config file:

args: -machine type=q35,kernel_irqchip=on

Another way to fix it is to roll back to the QEMU 3 machine model:

machine: pc-q35-3.1

Details:

https://forum.proxmox.com/threads/vm-w-pcie-passthrough-not-working-after-upgrading-to-6-0.56021/