[SOLVED] AMD GPU Passthrough

[summerside1967]

Hi Everyone...

I am definitely at a loss to get the RX 6700 XT to passthrough. I've tried for weeks following a lot of suggestions and guides and in the end I always get a error code 43. I'm not sure where to drill down to find out what is causing the problem. I've followed the ultimate user guide for passthrough, vendor reset and various scripts.

In windows 10 pro vm, the gpu shows up after first boot. Install drivers, then reboot and the gpu shows code 43.

I have an asus proart z690 creator MB, Intel(R) Core(TM) i9-12900K and the GPU.

Any help to investigate the source of the problem would be very appreciated.

 

[leesteken]

43 is a very generic error unfortunately. The Radeon 6000 series should not need vendor-reset and is also not supported by vendor-reset. Have you tried disabling Resizable BAR or Smart Access Memory? Is this the only GPU in the system and is it used during system boot? Then you might need to use the initcall_blacklist=sysfb_init kernel parameter as a work-around.

 

[summerside1967]

Thanks for the help. Proxmox uses the igpu. Confirmed in the bios that Resizable BAR is disabled. Removed the vendor reset from the vm config.

Booted up the windows 10 VM. Then uninstall the amd gpu, rescanned for hardware changes. It appeared again with no errors. Once I reboot windows 10, the gpu has the code 43 again. Seems to always reappear when I reboot the vm

 

[summerside1967]

I’m still can’t figure out why the gpa keeps giving a code 43 after I restart a VM. I’ve looked at so many different posts and tried a lot of suggestions. Any other thoughts? The gpu shows up under devices…I can install the amd drivers but when I restart windows 10 the gpu has code 43.

Thanks again. Very appreciated

 

[leesteken]

Try passthrough with a recent Ubuntu VM (Live ISO?) and see if the GPU works there. If it does, then it's a Windows driver issues that you need to find a work-around for.
If we can't get it to work on Ubuntu then maybe it's an issue with that particular (type or brand of) card. Does it happen to be on this list on reddit/r/VFIO?

 

[summerside1967]

@leesteken thanks for your reply. Looks like my card is on the list…unfortunately it’s in the group “don’t work”. I have a Sapphire RX 6700 XT Nitro. Thanks again for your help. I have an older NVidia card I can use for now

 

[summerside1967]

An update and a question. I made sure in the bios that the igpu is the primary graphics. I then swapped out the amd rx 6700xt for an older nvidia card. Booted up proxmox but could not connect to the webgui for proxmox

Was proxmox host using the amd rx6700xt even though I blacklisted the gpu drivers? The instant I put the amd card back in i could access the webgui again

How can I force proxmox to use the igpu? Did I miss something in setting up pass through? I followed a few guides for setup

Thanks for your help

 

[leesteken]

Even when blacklisting amdgpu, there are other output drivers that can use the basic functionality of the AMD GPU like the framebuffer drivers.
Usually you select the primary GPU in the motherboard BIOS. In addition, modern Intel integrated graphics also requires that a display is connected (and powered on but not necessarily displaying that input, I think).

 

[summerside1967]

Even when blacklisting amdgpu, there are other output drivers that can use the basic functionality of the AMD GPU like the framebuffer drivers.
Usually you select the primary GPU in the motherboard BIOS. In addition, modern Intel integrated graphics also requires that a display is connected (and powered on but not necessarily displaying that input, I think).

Thanks. The proxmox setup is in the basement. I have a monitor connected to the igpu hdmi port. Confirmed bios has igpu set as primary. My cpu is an intel i9 processor. I also have an hdmi dummy plugged into the amd card.

Any process I can do to see what proxmox is using for graphics?

 

[leesteken]

Is the motherboard BIOS shown on the monitor connected to the integrated graphics? Does it show something like "press DEL to enter setup" when starting? I have no experience with your motherboard or recent Intel processors.

You could just do early binding of the 6700XT to vfio-pci, then nothing can use it before the VM starts.

You might want to check your 6700XT against this list. If it does not reset properly then you need the early binding and will have problems when you shutdown and restart the VM (warm reboot of the VM initiated from the inside might work).

 

[summerside1967]

Is the motherboard BIOS shown on the monitor connected to the integrated graphics? Does it show something like "press DEL to enter setup" when starting? I have no experience with your motherboard or recent Intel processors.

You could just do early binding of the 6700XT to vfio-pci, then nothing can use it before the VM starts.

You might want to check your 6700XT against this list. If it does not reset properly then you need the early binding and will have problems when you shutdown and restart the VM (warm reboot of the VM initiated from the inside might work).

Thanks again. Yep, the boot and bios shows up on the monitor connected to the igpu

I’ll try early binding later of the 6700xt later today and let you know the results

Thanks

 

[summerside1967]

So this is what I've done so far

1. Confirmed the bios is set for igpu primary graphics
2. Added amdgpu to the blacklist
3. Early binding of the 6700xt

The graphics card shows up with no errors in windows 10 pro fresh vm install. The moment I restart windows, after updates, I get the following messages

Sep 24 12:41:06 pve kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0xffff@0xffc
Sep 24 12:41:06 pve kernel: vmbr0: port 2(tap301i0) entered disabled state
Sep 24 12:41:06 pve kernel: vmbr0: port 2(tap301i0) entered disabled state
Sep 24 12:41:06 pve pvedaemon[4379]: VM 301 qmp command failed - VM 301 not running
Sep 24 12:41:06 pve pvestatd[4349]: VM 301 qmp command failed - VM 301 not running
Sep 24 12:41:06 pve pvedaemon[4380]: VM 301 qmp command failed - VM 301 not running
Sep 24 12:41:06 pve kernel: vfio-pci 0000:03:00.0: can't change power state from D3cold to D0 (config space inaccessible)
Sep 24 12:41:06 pve kernel: vfio-pci 0000:03:00.0: can't change power state from D3cold to D0 (config space inaccessible)
Sep 24 12:41:06 pve kernel: vfio-pci 0000:03:00.0: can't change power state from D3cold to D0 (config space inaccessible)
Sep 24 12:41:07 pve pvedaemon[4993]: start failed: QEMU exited with code 1
Sep 24 12:41:07 pve pvedaemon[4380]: <root@pam> end task UPIDve:00001381:00003D3C:632F4F40:qmstart:301:root@pam: start failed: QEMU exited with code 1
Sep 24 12:41:08 pve kernel: vfio-pci 0000:03:00.0: invalid power transition (from D3cold to D3hot)
Sep 24 12:41:08 pve kernel: vfio-pci 0000:03:00.1: can't change power state from D0 to D3hot (config space inaccessible)

There are mutiple lines for "Sep 24 12:41:06 pve kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0xffff@0xffc"

Windows does not restart and I have to do a proxmox reboot to get it to work again

Here is my grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio-pci.ids=1002:73df,1002:ab28 video=vesafbff video=efifbff video=simplefbff"

Any thoughts?

No error 43 codes

 

[leesteken]

So this is what I've done so far

1. Confirmed the bios is set for igpu primary graphics
2. Added amdgpu to the blacklist
3. Early binding of the 6700xt

If you do early binding then you don't need to blacklist amdgpu.
You might need a softdep amdgpu pre: vfio_pci to make sure vfio-pci loads first, when you do all this in the /etc/modprobe.d/ directory (which you didn't).

The graphics card shows up with no errors in windows 10 pro fresh vm install. The moment I restart windows, after updates, I get the following messages
Windows does not restart and I have to do a proxmox reboot to get it to work again

Here is my grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio-pci.ids=1002:73df,1002:ab28 video=vesafbff video=efifbff video=simplefbff"

You don't need to use video=...fb:off when using vfio-pci.ids=.... Also: there are Inline code-tags available on this forum.

Any thoughts?

Your GPU does not appear to reset properly, which is uncommon for the Radeon 6000 series but unfortunately not unheard of for 6700XT.
Therefore, you need to restart the Proxmox host every time you have shutdown the VM and want to restart it. And maybe even when doing a warm reboot from within the VM.

No error 43 codes

Glad to see that you got passthrough working. Sorry for the reset issues of your card. Maybe vendor-reset will support it in the future?
Maybe using another reset-mechanism might work: what is the output of cat "/sys/bus/pci/devices/0000:03:00.0/reset_method"?
Can you share the precise make and model of the GPU as a warning to others?

 

[summerside1967]

cat "/sys/bus/pci/devices/0000:03:00.0/reset_method"

The only output I had was "bus"

The gpu is:

Sapphire Nitro+ AMD Radeon RX 6700 XT Gaming OC 12GB GDDR6 HDMI / Triple DP (11306-01-20G)​

Just confirmed it is on the list of gpus that don't work

I can repurpose this card so no worries. I want to stick with AMD and can get a deal on a 6800 or 6900. No issues with either of these cards that I can see?

Last question. What's the best way to replace a pcie gpu. Before I started playing around with settings, I took the 6700xt out and replaced it with an older Nvidia card I had from an old build. Proxmox booted up and I could see the text login on the igpu monitor. The webgui though was not accessible.

Thanks again for your help

 

[leesteken]

Last question. What's the best way to replace a pcie gpu. Before I started playing around with settings, I took the 6700xt out and replaced it with an older Nvidia card I had from an old build. Proxmox booted up and I could see the text login on the igpu monitor. The webgui though was not accessible.

The PCI IDs change (usually by one) when you add or remove a PCI(e) device. This includes on-board device like the network device and some devices have more than one PCI ID. The name of the network device (in /etc/network/interfaces) is based on the PCI ID. Therfore, you often need fixup the network configuration (which you can't do remotely for obvious reasons) when adding or removing PCI(e) devices. There are several threads on this forum about that.

 

[summerside1967]

Thanks again for your help! Very appreciated

 

[summerside1967]

Another question. Read some of the posts about adding pci(e) express devices…think I have a good understanding.

Before I switch out the gpu, my question is will the hard drives I currently have passed through be effected.

I have open media vault as a VM installed with ten hard drives passed through. Will this gave an effect on the hdd passthrough after the gpu swap?

Thanks again

 

[leesteken]

The PCI ID's might shift when you add/remove/change PCIe devices. You might need to adjust the passthrough configuration of other VMs accordingly. You might also want to check the IOMMU groups to make sure (all) your passthrough devices are not suddenly grouped with a new device.

 

[summerside1967]

Thanks again for your help. Just a little nervous about messing up the open media vault vm.

So I passed through ten hard drives using commands like this for each drive:

qm set 100 -scsi1 /dev/disk/by-id/(serial number here without brackets)

I use the MB built in sata as well as a pci sata controller card

Here is my current IIOMMU grouping with the 6700xt still installed:

root@pve:~# lspci
00:00.0 Host bridge: Intel Corporation Device 4660 (rev 02)
00:01.0 PCI bridge: Intel Corporation Device 460d (rev 02)
00:01.1 PCI bridge: Intel Corporation Device 462d (rev 02)
00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c)
00:0a.0 Signal processing controller: Intel Corporation Device 467d (rev 01)
00:0e.0 RAID bus controller: Intel Corporation Volume Management Device NVMe RAID Controller
00:14.0 USB controller: Intel Corporation Device 7ae0 (rev 11)
00:14.2 RAM memory: Intel Corporation Device 7aa7 (rev 11)
00:15.0 Serial bus controller [0c80]: Intel Corporation Device 7acc (rev 11)
00:15.1 Serial bus controller [0c80]: Intel Corporation Device 7acd (rev 11)
00:15.2 Serial bus controller [0c80]: Intel Corporation Device 7ace (rev 11)
00:16.0 Communication controller: Intel Corporation Device 7ae8 (rev 11)
00:17.0 SATA controller: Intel Corporation Device 7ae2 (rev 11)
00:1a.0 PCI bridge: Intel Corporation Device 7ac8 (rev 11)
00:1b.0 PCI bridge: Intel Corporation Device 7ac0 (rev 11)
00:1c.0 PCI bridge: Intel Corporation Device 7ab8 (rev 11)
00:1c.2 PCI bridge: Intel Corporation Device 7aba (rev 11)
00:1c.3 PCI bridge: Intel Corporation Device 7abb (rev 11)
00:1d.0 PCI bridge: Intel Corporation Device 7ab0 (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a84 (rev 11)
00:1f.3 Audio device: Intel Corporation Device 7ad0 (rev 11)
00:1f.4 SMBus: Intel Corporation Device 7aa3 (rev 11)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 7aa4 (rev 11)
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 (rev ff)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab28 (rev ff)
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
07:00.0 Ethernet controller: Aquantia Corp. Device 94c0 (rev 03)
08:00.0 Network controller: Intel Corporation Device 2725 (rev 1a)
09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03

Is it your experience that all the ids change? I just don't want to mess up the omv set up...the other vms don't matter. None of the vm boot at start

Thanks again

 

[leesteken]

You are passing through (partitions of) drives, which is not related to PCIe passthrough and I expect no interference.

Note that you are not showing the IOMMU groups, you are showing the PCI IDs of the devices.

My experience is that some PCI IDs change because the motherboard often assign them sequentially and removing a device can therefore cause devices with a higher PCI ID to shift. 04:00.0 might become 03:00.0 or even 01:00.0 in your case for example.