[SOLVED] AMD GPU Passthrough recently stopped working

[scyto]

error writing '1' to '/sys/bus/pci/devices/0000:0b:00.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:0b:00.0', but trying to continue as not all devices need a reset

this can probably be ignored, i have this all the time on my VM with pasthrough - it is not an idication of an actuall issue as not all devices need or support a reset

so unless you know your device does need this i would suggest looking elsehwere for your a cause of your issues and ignore this message (for example i get these for my SATA AHCI controllers i pass through)

 

[kobold81]

I haven´t played on my win11 for three months and after all the upgrading kernel and pve versions in between, passthrough ended with a freeze of the whole server. Ubuntu and batocera still work with the old config, had to be something windows related therefore.
Culprit was the AMDGopDriver for audio, when I removed that, I could boot win11 with passed through graphics,

I found in an old thread on the proxmox forum that someone passed all things related to audio to his win11 to make passthrough work at all for him at that time and I just tried that (even if that was nod needed before on my machine):

root@k8plus:~# lspci -nn | grep -e 'Audio'
c6:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
c6:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63)
c6:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]

The hostpci0 & 1 were there before, I just neeed to add the hostpci 2 & 3 and it just worked. I also use as machine: pc-q35-9.2+pve1,viommu=virtio

hostpci0: 0000:c6:00.0,pcie=1,romfile=vbios_7xxx.bin,x-vga=1
hostpci1: 0000:c6:00.1,pcie=1,romfile=AMDGopDriver.rom
hostpci2: 0000:c6:00.5,pcie=1
hostpci3: 0000:c6:00.6,pcie=1

 

[machone]

I haven´t played on my win11 for three months and after all the upgrading kernel and pve versions in between, passthrough ended with a freeze of the whole server. Ubuntu and batocera still work with the old config, had to be something windows related therefore.
Culprit was the AMDGopDriver for audio, when I removed that, I could boot win11 with passed through graphics,

I found in an old thread on the proxmox forum that someone passed all things related to audio to his win11 to make passthrough work at all for him at that time and I just tried that (even if that was nod needed before on my machine):

root@k8plus:~# lspci -nn | grep -e 'Audio'
c6:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
c6:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor [1022:15e2] (rev 63)
c6:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]

The hostpci0 & 1 were there before, I just neeed to add the hostpci 2 & 3 and it just worked. I also use as machine: pc-q35-9.2+pve1,viommu=virtio

hostpci0: 0000:c6:00.0,pcie=1,romfile=vbios_7xxx.bin,x-vga=1
hostpci1: 0000:c6:00.1,pcie=1,romfile=AMDGopDriver.rom
hostpci2: 0000:c6:00.5,pcie=1
hostpci3: 0000:c6:00.6,pcie=1

Thanks for the idea, but, no change unfortunately. This seems to be for passing through an iGPU, right?
Anyway I pulled the ID's for the other audio controllers on my machine, passed them through, even added them to vfio.conf etc so they don't get claimed by the host, rebooted and started the VM - same result.

 

[machone]

Okay, this is wild but turning off resizable BAR in my BIOS has me booting into a VM! I left above-4G decoding enabled.

If it's helpful I can post all of my configs but basically I have no kernel parameters, I added all of the GPU ID's into VFIO's options (and the GPU-related ones have disable_vga appended), there are no blacklisted drivers but softdep/early binding is enabled, and that's about it.

 

[machone]

As per https://www.reddit.com/r/VFIO/comme...8&utm_medium=web2x&context=3&utm_source=share and https://www.reddit.com/r/VFIO/comme...8&utm_medium=web2x&context=3&utm_source=share, I manually set my BAR via udev rules.

 

[machone]

@leesteken thanks for all your help, you got me most of the way there. Do you have any insight into why the Resizable BAR BIOS option would prevent passthrough from working properly? What is really odd to me is, forget Windows, the Ubuntu VM wouldn't output anything either with Resizable BAR enabled.

 

[leesteken]

Do you have any insight into why the Resizable BAR BIOS option would prevent passthrough from working properly?

Dynamically changing the BAR size(s) is not supported by VFIO (QEMU/KVM) but the driver inside the VM expects to be able to change the BAR size(s) (which I think also involves other parts of the PCIe infrastructure). That's why for the first couple of years we had to disable it when using PCI(e) passthrough.

Currently, VFIO does support changing the BAR size(s) statically before starting the VM: https://angrysysadmins.tech/index.p...eable-bar-rebar-in-your-vfio-virtual-machine/ . I don't know whether VFIO will support/emulate dynamic BAR size changes in the future.