AMD Radeon GPU Passthrough - Infamous reset bug

P

Patryk122PL

Member
Nov 22, 2023
5
0
6
Hello,

I'm not going to repeat what other users said / recommended - there are plenty of tutorials that more or less can get your AMD gpu working for VM passthrough.
My problem lays with very old and still preset in 9000 radeon gpu series issue (9070 xt specifically) - reset bug.

The best resources about this I found so far are those 2 github pages:
1. RX 9070 XT (RDNA4) passthrough on Proxmox — self-healing hookscript setup
2. Single-GPU Passthrough: AMD RX 9070 XT (Navi 48 / RDNA 4) on CachyOS

Ok, so i got 9070 xt working.. I'm using a slightly tweaked hook scripts from hughesyadaddy .. I can reliably pass gpu to vm and back to proxmox if.. (and that is a very big IF):

1. The hookscript unbinds GPU in pre-stop stage - so before VM process exits
2. I'm NOT using shutdown or restart procedures from within Windows VM itself.

From what I observed as soon as vfio-pci resets my gpu and this is automatic step after VM exit - my card becomes unstable and trying to bind it to amdgpu driver is successful only randomly. I posted more details here: Discussion question - more problems... #1

I would like to somehow block this automatic PCI reset procedure in proxmox or more specifically in QEMU.

Is there a way to disable or better yet exclude certain devices so that vfio-pci doesn't reset it on VM start and stop?
 
Best way would probably either be:
1.) AMD fixing the reset mechanism. Which they probably won't due to this being a Consumer GPU.
2.) Someone adding a PCI subsystem quirk for the card (Linux kernel code), as a workaround attempt.
 
Ok.. after a while of searching and poking around with various "hacks" I think my last resort is to indeed modify linux kernel.

So here's a quick test patch:

In 'drivers/pci/quirks.c':
C: 
/*
 * Test patch for AMD Radeon 9070 issues after bus reset.
 */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0x7550, quirk_no_bus_reset);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0xab40, quirk_no_bus_reset);

I have a trouble successfully compiling pve-kernel (stuck on ABI check) and at this moment I have no space for a dedicated dev VM with testing packages and working build toolchain.

Can someone with a working setup apply this patch and post precompiled .deb package?
 
Since Linux kernel 6.8 (recent Proxmox versions), you can read the available reset methods from /sys/bus/pci/devices/0000:XX:YY.0/reset_method. And just like vendor-reset requires device_specific, you can write to it (using the correct XX and YY for your GPU) to select a specific reset method.
The again, most RX 9070 should just work, even when amdgpu was loaded, on Proxmox kernel 7.0+ in my experience, which is limited to reading the forum, the internet in general and owning a 9070 GRE).

EDIT: did not realize you already tried everything, as you did say, and I cannot help with custom kernels, sorry. Let me know if you prefer me deleting this post.
 
Last edited:
Don't exactly know what you have in mind with that last sentence.. I explained in #1 post that I can reliably passthrough 9070 to a Guest VM and back to Proxmox as many times as I like if via hookscript I unbind it from a VM in pre-stop stage. Any other sequence or shutting down Win VM from start menu (inside guest OS) will result in VFIO resetting my GPU while it's bound to vfio-pci driver and this will freeze amdgpu driver in almost all cases (it then can't re-initialize the card - and a few moments later my host completely reboots).

I've seen the proposed solution; only valid reset_method is "bus" or "" (empty). In logs I can see amdgpu driver disabling every reset method and yet the GPU still is being reset by vfio-pci on VM start and stop.

The only "hack" I didn't try is the custom kernel.. but so far no luck in compiling.
 
No worries, thank you for attempting to help me.
From what I've gathered scattering the internet for the past month is that this issue seems to be present in certain gpu vendors so not always a specific model.

Anyway.. in my case **I HAVE TO** bind my card to amdgpu driver before it's reset by vfio, otherwise 9/10 times it will hang and take my host with it.
 
You must log in or register to reply here.
Top Bottom
Back