GPU Passthrough with RX 6900XT

[bishopg12]

I'm trying to set up GPU passthrough with an RX 6900XT. I followed the guide here: https://pve.proxmox.com/wiki/Pci_passthrough but i'm unable to detect the GPU in the guest OS. Here is what I tried.


Step 1:
As I was unsure if I'm using grub or systemd-boot, I modified both /etc/default/grub and /etc/kernel/cmdline:

Modifications to /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

Modifications to /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt

I ran both udate-grub and proxmox-boot-tool refresh after modifcation.

Step 2:
I verified that IOMMU is enabled:

root@beatrice:~# dmesg | grep -e DMAR -e IOMMU
[ 1.239448] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 1.241134] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 1.241834] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 5.680671] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

Step 3:
Modified /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Step 4:
Verified IOMMU Interrupt Remapping

root@beatrice:~# dmesg | grep 'remapping'
[ 1.241139] AMD-Vi: Interrupt remapping enabled

Step 5:
Verified IOMMU Isolation.

root@beatrice:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:03:03.0
/sys/kernel/iommu_groups/35/devices/0000:10:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/25/devices/0000:07:00.0
/sys/kernel/iommu_groups/15/devices/0000:02:00.0
/sys/kernel/iommu_groups/33/devices/0000:0f:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/23/devices/0000:04:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/31/devices/0000:0e:00.2
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:03:09.0
/sys/kernel/iommu_groups/21/devices/0000:0a:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:0d:00.0
/sys/kernel/iommu_groups/18/devices/0000:03:04.0
/sys/kernel/iommu_groups/36/devices/0000:10:00.3
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
/sys/kernel/iommu_groups/26/devices/0000:08:00.0
/sys/kernel/iommu_groups/16/devices/0000:03:02.0
/sys/kernel/iommu_groups/34/devices/0000:10:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/24/devices/0000:06:00.0
/sys/kernel/iommu_groups/24/devices/0000:06:00.1
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/32/devices/0000:0e:00.3
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/22/devices/0000:0b:00.0
/sys/kernel/iommu_groups/22/devices/0000:03:0a.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/30/devices/0000:0e:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:09:00.0
/sys/kernel/iommu_groups/20/devices/0000:09:00.3
/sys/kernel/iommu_groups/20/devices/0000:09:00.1
/sys/kernel/iommu_groups/20/devices/0000:03:08.0
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/29/devices/0000:0e:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:03:05.0
/sys/kernel/iommu_groups/37/devices/0000:10:00.4
/sys/kernel/iommu_groups/9/devices/0000:00:07.1
/sys/kernel/iommu_groups/27/devices/0000:0c:00.0

Step 6:
I found the device and vendor id as follows:

root@beatrice:~# lspci -n -s 0e:00
0e:00.0 0300: 1002:73bf (rev c0)
0e:00.1 0403: 1002:ab28
0e:00.2 0c03: 1002:73a6
0e:00.3 0c80: 1002:73a4

I then created the file /etc/modprobe.d/vfio.conf with the following:
options vfio-pci ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4 disable_vga=1

I then blacklisted radeon:
blacklist radeon

I made sure the guest was set up as follows:
machine: q35
bios: OVMF

I added the PCIe device 0000:0e:00.0 in the gui while selecting all functions, primary GPU, ROM-Bar, and PCI-Express.

Step 7:
I tried installing windows 10 pro, but the driver cannot find the device. I do see the devices in device manager but they are of unknown type. Am I missing something from the setup?

 

[leesteken]

Can you try with Primary GPU disabled, and just set Display to none? Primary GPU does a lot of things for NVidia cards but I have had better luck with AMD cards without it.

 

[rifaterdemsahin]

old settings >

new ones with the pcipress on
primiray GPU off

Card and 6900 xt

I have black listed
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

 

[rifaterdemsahin]

installed without mentioning the primary GPU
then installed the driver
than I plan telling as primary gpu

 

[rifaterdemsahin]

did not work
now display none ( close )
gpu not primary ( opens up ) ( as I know that works that might help)

 

[bilgealp015]

i have got same problem.

 

[buzzmountain]

Any resolution on this? I have the same issue - new to proxmox and also using an old box used for gaming (ryzen 9 5950xt; 128gb ram; radeon 6900xt). Please yell back

 

[johnsonvenezuellas90]

When setting up a GPU passthrough with an RX 6900XT, using a custom resolution utility on Windows 11 is often the only way to fix display detection issues in your VM. This tool allows you to manually inject the correct EDID data into the registry, ensuring your virtual machine recognizes the high-performance capabilities of your 6900XT. It's an essential step for bypassing driver-related black screens and achieving native monitor refresh rates in a virtualized environment.

 

[buzzmountain]

I'm not using Windows 11 in my Proxmox server

 

[uzumo]

In this thread, the original poster is asking about a situation where Windows recognizes a device in Device Manager but cannot install the driver.

You are saying that without using Windows, you experience the same problem as the original poster on Windows.

What does that mean?

 

[buzzmountain]

In this thread, the original poster is asking about a situation where Windows recognizes a device in Device Manager but cannot install the driver.

You are saying that without using Windows, you experience the same problem as the original poster on Windows.

What does that mean?

One of us is misreading the original post. What I discern is OP is trying to get PCI passthrough to work in Proxmox with a Radeon 6900XT. When results gave conflicting messages, OP then tried to install Windows, in what I assume a VM, and the device is seen in device manager, but the PCI passthrough in Proxmox is not working. I have no intention of running ANY windows OS on my Proxmox server, but I DO need the PCI passthrough for Radeon 6900XT for Frigate.