PCI Passthrough fail when booting with 5.11.22-7-pve

[JOduMonT]

Hi; I'm pleased to report a bug
while everything is working great with the kernel 5.11.22-5-pve none of my VM which have a VGA Passthrough boot with the kernel 5.11.2207-pve.

NOTE: all VM with PCI Passthrough remains working if I boot with the old kernel.

the error is: TASK ERROR: Cannot bind 0000:01:00.0 to vfio
which is an old Radeon

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev ef) (prog-if 00 [VGA controller])
        Subsystem: XFX Pine Group Inc. Radeon RX 570
        Flags: bus master, fast devsel, latency 0, IRQ 140, IOMMU group 1
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at e0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at e000 [size=256]
        Memory at ef100000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at ef140000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] Physical Resizable BAR
        Capabilities: [270] Secondary PCI Express
        Capabilities: [2b0] Address Translation Service (ATS)
        Capabilities: [2c0] Page Request Interface (PRI)
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [370] L1 PM Substates
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

which is used by 2 VM (not simultaneously of course
1 Windows 10 Pro and 1 Ubuntu 20.04 LTS

agent: 1
balloon: 2048
bios: ovmf
boot: order=virtio0;ide2
cores: 4
efidisk0: nvme:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:01:00,pcie=1,x-vga=1
ide2: none,media=cdrom
machine: pc-q35-6.0
memory: 8192
name: windows
net0: virtio=CA:66:71:93:4E:63,bridge=vmbr0
net1: virtio=22:73:BA:64:7E:F9,bridge=vmbr1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=c66952c0-f6a3-4154-bab3-61fd1fd10e6a
sockets: 1
tpmstate0: nvme:vm-110-disk-1,size=4M,version=v2.0
usb0: host=1-13
usb1: host=1-12
vga: none
virtio0: nvme:vm-110-disk-2,cache=writeback,discard=on,size=64G
vmgenid: 79ff4d87-41da-4bb4-ba1f-f74f48a739e9

the host is an Intel i5 E3-1200 v5/E3-1500 v5/6th on a motherboard with z170 chipset.
All the proxmox setup was running fine since few years until the update

 

[happyfredo]

Hello,

Different mistake here but i think it's linked.
VGA Passthrough (nvidia) for MACOS (OPENCORE) don't work anymore with 5.11.2207-pve and it work fine with kernel 5.11.22-5-pve. Black screen but no error (THE VM is UP)
Strange things, it work fine WITH WINDOWS 10.

 

[Duanra08]

Hello,
I have the same problem as @JOduMonT.
I have configured my grub to boot with the old kernel to resolve the PCI-Passtrought problem.

 

[Duanra08]

Hello, after the migration to promox 7.1-5 and the kernel 5.13.19-1
I have again the problem...

 

[leesteken]

I noticed that the amdgpu driver no longer releases the GPU to vfio-pci since kernel 5.11.22-7. My work-around was binding the GPU to vfio-pci early and also use video=efibf:off.

 

[Duanra08]

I have restart Proxmox 7.1-5 with the old kernel 5.11.22-5, proxmox relaunch and my windows VM with PCI-Passtrought are available!

 

[Duanra08]

I have compared the rseult of "lshw -C display" with the kernel 5.11.22-5 and 5.13.19-1

kernel 5.11.22-5 root@pve:/boot/grub# lshw -C display *-display description: VGA compatible controller product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] vendor: Advanced Micro Devices, Inc. [AMD/ATI] physical id: 0 bus info: pci@0000:01:00.0 version: e1 width: 64 bits clock: 33MHz capabilities: pm pciexpress msi vga_controller bus_master cap_list rom configuration: driver=vfio-pci latency=0 resources: irq:146 memory:e0000000-efffffff memory:f0000000-f01fffff ioport:e000(size=256) memory:f7e00000-f7e3ffff memory:c0000-dffff kernel 5.13.19-1 root@pve:~# lshw -C display *-display UNCLAIMED description: VGA compatible controller product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] vendor: Advanced Micro Devices, Inc. [AMD/ATI] physical id: 0 bus info: pci@0000:01:00.0 version: e1 width: 64 bits clock: 33MHz capabilities: pm pciexpress msi vga_controller bus_master cap_list configuration: latency=0 resources: memory:e0000000-efffffff memory:f0000000-f01fffff ioport:e000(size=256) memory:f7e00000-f7e3ffff memory:c0000-dffff

We have diffrence in:
- capabilities: no rom
- configuration: no driver=vfio-pci
- resources: no irq=146

I don't know how to add this parameters in the new kernel....

 

[leesteken]

I have compared the rseult of "lshw -C display" with the kernel 5.11.22-5 and 5.13.19-1

kernel 5.11.22-5 root@pve:/boot/grub# lshw -C display configuration: driver=vfio-pci latency=0 kernel 5.13.19-1 configuration: latency=0

We have diffrence in:
- capabilities: no rom
- configuration: no driver=vfio-pci
- resources: no irq=146

I don't know how to add this parameters in the new kernel....

It looks like the differences come from loading vfio-pci for the device in 5.11.22-5 and not doing so in 5.13.19-1. I don't understand what you are trying to show here.
Also, I don't understand what you are asking about parameters, sorry.