AMD GPU Passthrough

[fwr]

Hello everyone,

I've been trying to get GPU passthrough working for a while now, but without success.

Here is my system:

MSI MAG B650 Tomahawk WIFI (Socket AM5) AMD B650 ATX DDR5 Wi-Fi 6E Motherboard - Black
XFX Radeon RX 7800 XT Merc 319
AMD RYZEN 9 7900X

I've followed multiple tutorials but still can't figure out why it's not working or what else I can try. I have IOMMU and virtualization enabled in my motherboard settings. Could my hardware be incompatible?

I came across a tutorial suggesting that I need to download a ROM file, but I couldn't find one for my GPU on the recommended website. (techpowerup)

Here are my latest settings:


root@fwr:~# efibootmgr -v
BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0008,0007,0009,000A,000B
Boot0000* proxmox HD(2,GPT,ca902a38-8c27-4cb2-af25-1cd74536ddec,0x800,0x200000)/File(\EFI\proxmox\shimx64.efi)
Boot0007* UEFI: VendorCoProductCode 2.00, Partition 2 PciRoot(0x0)/Pci(0x8,0x1)/Pci(0x0,0x3)/USB(1,0)/HD(2,GPT,cc204a36-7205-49c2-80bd-642a2e65b1fd,0x224,0x4000)..BO
Boot0008* UEFI OS HD(2,GPT,ca902a38-8c27-4cb2-af25-1cd74536ddec,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
Boot0009* UEFI:CD/DVD Drive BBS(129,,0x0)
Boot000A* UEFI:Removable Device BBS(130,,0x0)
Boot000B* UEFI:Network Device BBS(131,,0x0)

root@fwr:~# dmesg |grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.182296] AMD-Vi: Using global IVHD EFR:0x246577efa2254afa, EFR2:0x0
[ 0.486953] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.488068] AMD-Vi: Extended features (0x246577efa2254afa, 0x0): PPR NX GT [5] IA GA PC GA_vAPIC
[ 0.488073] AMD-Vi: Interrupt remapping enabled
[ 0.694658] AMD-Vi: Virtual APIC enabled
[ 0.694903] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

root@fwr:~# dmesg | grep 'remapping'
[ 0.488073] AMD-Vi: Interrupt remapping enabled

root@fwr:~# dmesg | grep -i vfio
[ 3.655949] VFIO - User Level meta-driver version: 0.3
[ 3.660745] vfio-pci 0000:03:00.0: vgaarb: deactivate vga console
[ 3.660750] vfio-pci 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 3.660907] vfio_pci: add [1002:747e[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.660924] vfio-pci 0000:12:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=nonewns=none
[ 3.661033] vfio_pci: add [1002:164e[ffffffff:ffffffff]] class 0x000000/00000000

root@fwr:~# lspci -nnk | grep 'AMD'
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Root Complex [1022:14d8]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge IOMMU [1022:14d9]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0 [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1 [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2 [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3 [1022:14e3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4 [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5 [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6 [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7 [1022:14e7]
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 11)
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 11)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 32 [Radeon RX 7700 XT / 7800 XT] [1002:747e] (rev c8)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio [1002:ab30]
05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
06:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:0b.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
06:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
10:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller [1022:43f7] (rev 01)
11:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
12:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c2)
12:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
12:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
12:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b6]
12:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b7]
12:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
13:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
_______________________________________________________________________________________________________________
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init"
GRUB_CMDLINE_LINUX=""
___________________________________________________________________________________________________________________

GNU nano 7.2 /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
____________________________________________________________________________________________________________________

GNU nano 7.2 /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
_____________________________________________________________________________________________________________________

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
#blacklist nvidiafb
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_codec_hdmi
blacklist snd_hda_intel
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist radeon
blacklist amdgpu
_________________________________________________________________________________________________________________________

GNU nano 7.2 /etc/modprobe.d/vfio-pci.conf
options vfio-pci ids=1002:747e,1002:164e disable_vga=1
___________________________________________________________________________________________________________________________





As you can see, when I try to add a PCI device to my Windows VM, my GPU does not appear.

I would really appreciate any help, as I'm not sure if I'm doing something wrong or missing a step. I hope this information is sufficient, but please let me know if there's anything else I should provide.

 

[guruevi]

For the few AMD GPU cards we have, we have to specify the PCI ID, in your case after amd_iommu=on iommu=pt vfio-pci.ids=1002:…,1002:…

That way the OS driver doesn’t accidentally load (my documentation suspected that AMD GPU won’t go into VFIO once the card has been activated by the OS this was under PVE7) - I’ve read since about a reset-bug in AMD hardware but can’t remember the details.

Then there is the next hurdle, on consumer hardware your PCI addresses may overlap, if your VM won’t boot, crashes or the host crashes or has other weird behavior, AMD CPU PCIe is buggy and has sets of hardware in the same VFIO address space, there is a software workaround for that.

If your machine boots in Windows but won’t see the card, test with a Linux boot disk and do lspci to see if the card pass through is working. From what I read elsewhere, the consumer GPU may only work on Linux with the open source amdgpu driver. That is where the flashing of your card comes in, back in the day that was possible as consumer and pro cards were largely ‘the same’ but AMD hasn’t brought a Radeon Pro in the open market for 2 years (you can rent their current Pro hardware on Azure). I’m not sure there is anything close to your card in a Radeon Pro jacket.

 

[fwr]

For the few AMD GPU cards we have, we have to specify the PCI ID, in your case after amd_iommu=on iommu=pt vfio-pci.ids=1002:…,1002:…

That way the OS driver doesn’t accidentally load (my documentation suspected that AMD GPU won’t go into VFIO once the card has been activated by the OS this was under PVE7) - I’ve read since about a reset-bug in AMD hardware but can’t remember the details.

Then there is the next hurdle, on consumer hardware your PCI addresses may overlap, if your VM won’t boot, crashes or the host crashes or has other weird behavior, AMD CPU PCIe is buggy and has sets of hardware in the same VFIO address space, there is a software workaround for that.

If your machine boots in Windows but won’t see the card, test with a Linux boot disk and do lspci to see if the card pass through is working. From what I read elsewhere, the consumer GPU may only work on Linux with the open source amdgpu driver. That is where the flashing of your card comes in, back in the day that was possible as consumer and pro cards were largely ‘the same’ but AMD hasn’t brought a Radeon Pro in the open market for 2 years (you can rent their current Pro hardware on Azure). I’m not sure there is anything close to your card in a Radeon Pro jacket.

It does see the card on linux, even on proxmox but for some reason on passthrough it does not work. I'm still trying to figure it out, i ust don;t ahve too much time for it at this moment. Thanks!

 

[guruevi]

If it works on a Linux guest, but the same config doesn’t work on a Windows guest, then the problem is the OS, not Proxmox. As I said, when I tested it, consumer AMD cards didn’t work in VM, even if I got them to pop up by basically instructing KVM to load a custom firmware ROM (the Pro version of the card) into the address space, they would still have error 13 in the hardware list. Only the open source driver works under Linux, you need a Pro card for the Windows driver to work.

This is for Arch, but you’ll see the ROM file reference:
https://github.com/mike11207/single-gpu-passthrough-amd-gpu/blob/main/README.md