RX 7900xtx Passthrough Issues

S

Salizar

New Member
Jul 31, 2023
4
0
1
I had a working passthrough using my old GPU (Nvidia GTX 970) but after upgrading to an AMD GPU I can't seem to get it working. I may have missed something obvious after making so many unnecessary changes to my configuration trying to get this to work. I've trimmed the fat since.

I'm currently using PVE 8, Kernel 6.2.16-5-pve

/etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet"

/etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

lspci -nnk
Code: 
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 10)
        Kernel driver in use: pcieport
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 10)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Kernel driver in use: pcieport
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:0e3b]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: i2c-designware-pci
        Kernel modules: i2c_designware_pci
(Full output at the end if needed)

/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=1002:744c,1002:ab30,1002:7446,1002:7444 disable_vga=1

/etc/pve/qemu-server/141.conf
Code: 
affinity: 0-7
balloon: 16384
bios: ovmf
boot: order=ide2;scsi0;net0
cores: 8
cpu: host
cpuunits: 1000
efidisk0: local-lvm:vm-141-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00,pcie=1
ide2: none,media=cdrom
machine: pc-q35-8.0
memory: 32768
meta: creation-qemu=8.0.2,ctime=1690765557
name: Winizar-xtx
net0: virtio=46:F5:3D:F2:D8:1D,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: local-lvm:vm-141-disk-1,aio=native,cache=directsync,iothread=1,size=100G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=368678e1-1293-4f55-8c64-4c58fd8ca346
sockets: 1
tpmstate0: local-lvm:vm-141-disk-2,size=4M,version=v2.0
vga: none
vmgenid: 367c9da7-341e-4516-b28a-678e1d58c856

I don't currently have anything blacklisted.

I'm mostly uncertain about "Navi 10 XL Upstream/Downstream Port of PCI Express Switch" since I don't remember Nvidia having anything similar. This card also has 4 devices (video, audio, USB, serial) instead of the 2 I've seen previously (video and audio) so I don't know if that could be causing an issue.

My results are Windows 10 fails to boot and enters a repair mode before eventually shutting down. At least, if I set the display to default and view noVNC that's what I see. Normally I try to rdp into Windows when I have the display set to none, but Windows doesn't boot so I can't connect.

Please let me know if I can provide any more info. Hopefully I just made a silly mistake somewhere. Thanks.







lspci -nnk
Code: 
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14d8]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14d9]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: pcieport
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: pcieport
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Subsystem: Device [7d69:1462]
        Kernel driver in use: pcieport
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
        Subsystem: Device [7d69:1462]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        Subsystem: Micro-Star International Co., Ltd. [MSI] FCH SMBus Controller [1462:7d69]
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Micro-Star International Co., Ltd. [MSI] FCH LPC Bridge [1462:7d69]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev 10)
        Kernel driver in use: pcieport
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (rev 10)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        Kernel driver in use: pcieport
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:0e3b]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: i2c-designware-pci
        Kernel modules: i2c_designware_pci
04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
05:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
06:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013] (rev 01)
        Subsystem: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013]
        Kernel driver in use: nvme
        Kernel modules: nvme
07:00.0 Ethernet controller [0200]: Aquantia Corp. AQC113CS NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:94c0] (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] AQC113CS NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1462:7d69]
        Kernel driver in use: atlantic
        Kernel modules: atlantic
09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0a:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:3328]
        Kernel driver in use: pcieport
0b:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013] (rev 01)
        Subsystem: Kingston Technology Company, Inc. FURY Renegade NVMe SSD [2646:5013]
        Kernel driver in use: nvme
        Kernel modules: nvme
0c:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
        Subsystem: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
        Kernel driver in use: mt7921e
        Kernel modules: mt7921e
0d:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
        Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:1060]
        Kernel driver in use: ahci
        Kernel modules: ahci
11:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
12:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1062]
        Kernel driver in use: ahci
        Kernel modules: ahci
13:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
14:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
        Subsystem: ASMedia Technology Inc. Device [1b21:1062]
        Kernel driver in use: ahci
        Kernel modules: ahci
15:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:3241]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:4480]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
16:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Raphael [1462:7d69]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
16:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Rembrandt Radeon High Definition Audio Controller [1462:7d69]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
16:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
        Subsystem: Micro-Star International Co., Ltd. [MSI] VanGogh PSP/CCP [1462:7d69]
        Kernel driver in use: ccp
        Kernel modules: ccp
16:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
16:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
16:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
        DeviceName: Realtek ALC1220
        Subsystem: Micro-Star International Co., Ltd. [MSI] Family 17h/19h HD Audio Controller [1462:ed69]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
17:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d69]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
 
Salizar said:
/etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:744c,1002:ab30,1002:7446,1002:7444 disable_vga=1
Click to expand...
You might want to add a softdep amdgpu pre: vfio-pci to make sure vfio-pci is loaded before amdgpu and can claim those devices.
Salizar said:
I'm mostly uncertain about "Navi 10 XL Upstream/Downstream Port of PCI Express Switch" since I don't remember Nvidia having anything similar. This card also has 4 devices (video, audio, USB, serial) instead of the 2 I've seen previously (video and audio) so I don't know if that could be causing an issue.
Click to expand...
It looks very similar to a 6950XT with a USB-C output and should not be a problem for passthrough.

Did you make sure to disable Resizable BAR (or whatever it is called by your motherboard BIOS) as it's not (yet) supported?
 
leesteken said:
You might want to add a softdep amdgpu pre: vfio-pci to make sure vfio-pci is loaded before amdgpu and can claim those devices.

It looks very similar to a 6950XT with a USB-C output and should not be a problem for passthrough.

Did you make sure to disable Resizable BAR (or whatever it is called by your motherboard BIOS) as it's not (yet) supported?
Click to expand...

I've edited vfio.conf
Code: 
softdep amdgpu pre: vfio-pci
options vfio-pci ids=1002:744c,1002:ab30,1002:7446,1002:7444 disable_vga=1
So before I boot any VM that uses the GPU lspci -nnk returns
Code: 
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:0e3b]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: i2c-designware-pci
        Kernel modules: i2c_designware_pci

In BIOS I have Resizable BAR Disabled. Set to UEFI and IOMMU is Enabled. Those were the only BIOS changes that I've made.


On a sidenote I've setup a Linux VM with a similar configuration to Windows. It boots with the GPU passed through, but I don't get a screen on my monitor. Most I did with that was SSH into it and see the GPU was connected.
 
Salizar said:
I've edited vfio.conf
Code: 
softdep amdgpu pre: vfio-pci
options vfio-pci ids=1002:744c,1002:ab30,1002:7446,1002:7444 disable_vga=1
So before I boot any VM that uses the GPU lspci -nnk returns
Code: 
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:0e3b]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: i2c-designware-pci
        Kernel modules: i2c_designware_pci
Click to expand...
You probably also need softdeps for snd_hda_intel, xhci_pci and i2c_designware_pci, just like you did for amdgpu. Sorry for not seeing this before.
Salizar said:
In BIOS I have Resizable BAR Disabled. Set to UEFI and IOMMU is Enabled. Those were the only BIOS changes that I've made.
Click to expand...
It's not like I know what the default BIOS settings for your motherboard and BIOS version are. Maybe you need to change other as well. I really don't know, sorry.
Salizar said:
On a sidenote I've setup a Linux VM with a similar configuration to Windows. It boots with the GPU passed through, but I don't get a screen on my monitor. Most I did with that was SSH into it and see the GPU was connected.
Click to expand...
Did you set Display to None (vga: none)? Try booting your VM with the most recent Ubuntu Live installer ISO (without installing) and I would expect to see some output from the GPU on a physical display. However, it might be that the GPU is too new for most Linux live ISOs...
 
leesteken said:
You probably also need softdeps for snd_hda_intel, xhci_pci and i2c_designware_pci, just like you did for amdgpu. Sorry for not seeing this before.
Click to expand...
Added for the other devices so I get,
Code: 
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:744c] (rev c8)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] [1002:0e3b]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab30]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7446]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7444]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
        Kernel driver in use: vfio-pci
        Kernel modules: i2c_designware_pci
leesteken said:
It's not like I know what the default BIOS settings for your motherboard and BIOS version are. Maybe you need to change other as well. I really don't know, sorry.
Click to expand...
Those were just the main settings I've seen come up. I should have included it with my post.
leesteken said:
Did you set Display to None (vga: none)? Try booting your VM with the most recent Ubuntu Live installer ISO (without installing) and I would expect to see some output from the GPU on a physical display. However, it might be that the GPU is too new for most Linux live ISOs...
Click to expand...
Displays set to None. Downloaded and tried booting latest Ubuntu (23.04) ISO. Screen turns black, after about a minute the VMs marked with a yellow warning triangle and when I mouse over it I'm getting Status: Internal-Error.
I just downloaded Ubuntu so I tried removing the gpu and setting the display to default afterwards, can confirm it works fine that way at least.

Not sure if this is relevant but I get a few lines displayed by the GPU during Proxmox startup. I didn't think too much of them since I had them appearing on a previous setup. Imagine that's normal since it comes before 'loading initial ramdisk'.
Code: 
    Booting 'Proxmox VE GNU/Linux'

Loading Linux 6.2.16-5-pve ...
Loading initial ramdisk ...
_

Thanks for replying leesteken. I see your name pop up in a lot of these passthrough posts so I was hoping you'd show up haha.
 
Salizar said:
Thanks for replying leesteken. I see your name pop up in a lot of these passthrough posts so I was hoping you'd show up haha.
Click to expand...
You're welcome. (I did feel the need to disabled private messages because people kept asking for private help.)
Do you know of anyone having success with PCIe passthrough of the 7900XT(X)? Could be that AMD broke it again after the 6800-6950XT GPU that passthrough very well.
Salizar said:
Not sure if this is relevant but I get a few lines displayed by the GPU during Proxmox startup. I didn't think too much of them since I had them appearing on a previous setup. Imagine that's normal since it comes before 'loading initial ramdisk'.
Code: 
    Booting 'Proxmox VE GNU/Linux'

Loading Linux 6.2.16-5-pve ...
Loading initial ramdisk ...
_
Click to expand...
That could be a problem. Make sure to boot the system with another (integrated) GPU (by plugging in a display to make sure). Or try this work-around but then you don't see any boot messages. Since you appear to have integrated graphics, please use it for boot and Proxmox host console so the 7900XTX is not touched by anything before the VM starts.
 
leesteken said:
You're welcome. (I did feel the need to disabled private messages because people kept asking for private help.)
Do you know of anyone having success with PCIe passthrough of the 7900XT(X)? Could be that AMD broke it again after the 6800-6950XT GPU that passthrough very well.

That could be a problem. Make sure to boot the system with another (integrated) GPU (by plugging in a display to make sure). Or try this work-around but then you don't see any boot messages. Since you appear to have integrated graphics, please use it for boot and Proxmox host console so the 7900XTX is not touched by anything before the VM starts.
Click to expand...
Had to unplug display cables from the GPU before I turned on my machine. Could then plug them in before starting up the VM and it all worked properly.

I have another setup where I use a ROM file. It displays some text through the GPU before the VM starts so I'm hoping if I add a ROM file to this one it might work the same. I'll update the post if I get it working.

Thanks for your help leesteken!
 
Hello Salizar, Leesteken and others on this issue. I just installed myself some proxmox, went through following this whole page of setup for passthrough and it worked! But as soon as I boot my PC with the GPU passing through(I also had to unplug the monitors to make this work) My PC starts up with 99% of ram in use. And it's not slow, the instant the Proxmox host starts reading the ram to put into the web portal, it's already maxed out. I don't know what I did to cause this... My current settings are basically as you guys have, but with a few modifications:

my /etc/default/grub:

Code: 
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet amd_iommu=on iommu=pt vfio-pci.ids=1002:744c,1002:1478,1002:1479,1002:1479,148c:2422,1002:ab30 initcall_blacklisacklist=sysfb_init  default_hugepagesz=1G hugepagesz=1G hugepages=32 amdgpu.sg_display=0"
GRUB_CMDLINE_LINUX="textonly video=astdrmfb video=efifb:off"

my /etc/modules:

Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

and my /etc/modprobe.d/vfio.conf:

Code: 
softdep i2c_designware_pci pre: vfio-pci
softdep xhci_pci pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
softdep amdgpu pre: vfio-pci
options vfio-pci ids=1002:744c,1002:1478,1002:1479,1002:1479,148c:2422,1002:ab30 disable_vga=1

and when I use free -h to see what's happening, I get this:
Code: 
               total        used        free      shared  buff/cache   available
Mem:            31Gi        30Gi       634Mi        44Mi       371Mi       689Mi
Swap:          8.0Gi          0B       8.0Gi

I'm probably going to post this as it's own issue as well, but you guys seemed the most knowledgable on the rx7900xtx specifically.

Edit: It really just took pasting it out in here to see it. I recreated my proxmox from just these components and removed the three references to hugepages which seemed to clear that issue up right away. On to building my first VM that can actually see this GPU! Thanks for the help everyone!

Update: I ran some updates. The GPU passthrough is working much better now! Even when the uefi/bios boots up, and proxmox starts to display on the primary graphics card, the screen stays on with the text showing that proxmox was booting, and as long as the video gets redirected to the secondary graphics card before the VM boots, the Windows VM is able to hook my 7900 xtx without issue. I can reboot remotely, and the server, as well as the PC come back on as normal! I didn't open this problem, but someone seems to have fixed it for me!
Key points I would like to make are that a secondary Graphics output is needed. I have no onboard display, so I had to plug in an old asus gt710. This gives proxmox a place to display the IP address the management is available on, and then you can passthrough the GPU even if it was the primary GPU in bios.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom