Proxmox 8 - AMD 6800XT and Nvidia A2000 Dual GPU Passthrough

[rupint]

Hi this is more documentation for me and to provide help to fellow forum lurkers who might be stuck on the same configs.

## Intro
So following the normal guides to passthrough GPUs got the Nvidia GPU working fine in a plex ubuntu vm but the AMD would not work for a while, finally figured it out over this weekend.

## Flex
What I have in my machine (and what I do with it):
Proxmox 8.0.3 bookworm
AMD Ryzen 9 5950X (amd resizable bar and 4g decoding are both disabled in Bios necessary for VM to boot and also not supported inside vm so anyways a waste)
128 GB RAM
AMD 6800XT (Passthrough to Windows Gaming VM or MacOS Ventura VM can only boot one)
Nvidia A2000 (Passthrough to Plex Ubuntu)
10gig Networking connected to a NAS box with dump backups
3 Pcie4.0 SSDs of 2TB capacity each (native Raidz1 ZFS for Proxmox)
6 SATA ssds - 8TB(3)/4TB(3) (Passthrough to Truenas Scale) with a zpool made of z1+z1 vdevs
That was my flex on with the rest

Since I use ZFS in proxmox my cmdline is not in /etc/default/grub but in /etc/kernel/cmdline omit the starting part root=ZFS=rpool/ROOT/pve-1 boot=zfs when using grub. My /etc/kernel/cmdline:

root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt textonly quiet vfio_iommu_type1 allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction nofb nomodeset initcall_blacklist=sysfb_init

You must use initcall_blacklist=sysfb_init instead of video=efifb:off the later no longer works above 5.15 kernel (Proxmox 7.x) I believe

My config files mostly the /etc/modprobe.d/vfio.conf is where the magic happens:

root@superpve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
### donot blacklist amdgpu

root@superpve:~# cat /etc/modprobe.d/kvm.conf
cat: /etc/modprobe.d/kvm.conf: No such file or directory
### I deleted this file and moved the ignore_msrs line into vfio.conf the next file

root@superpve:~# cat /etc/modprobe.d/vfio.conf
options kvm ignore_msrs=1 ### moved from kvm.conf
options vfio-pci ids=10de:2531,10de:228e disable_vga=1 ### this is only the pci ids of the Nvidia the AMD ones are not included not sure what black magic this is
softdep amdgpu pre: vfio vfio_pci ### dont know what this means but doesnt work without it and basically I think forces the AMD gpu to use vfio driver

root@superpve:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
### as expected from generic passthrough guide

root@superpve:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
### as expected from generic passthrough guide

Use update-initramfs -k all -u && update-grub && pve-efiboot-tool refresh to refresh your boot config

Debug commands that helped me:

root@superpve:~# cat /proc/iomem | grep -e BOOTFB
root@superpve:~# should be no output
root@superpve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.105072] AMD-Vi: Unknown option - 'on'
[    0.249957] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[    0.864314] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.866486] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.866486] AMD-Vi: Extended features (0x58f77ef22294a5a, 0x0): PPR NX GT IA PC GA_vAPIC
[    0.866489] AMD-Vi: Interrupt remapping enabled
[    0.866813] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    4.280988] AMD-Vi: AMD IOMMUv2 loaded and initialized
root@superpve:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done;
root@superpve:~# above verifies everything has different iommu groups

Another onedmesg | grep -e "can't reserve [mem" but mine was this could not be reserved as shown below. Not sure why I have this error but everything seems to still be working.

root@superpve:~#  dmesg | grep -e "could not be reserved"
[    0.854747] system 00:04: [mem 0xfec00000-0xfec00fff] could not be reserved
[    0.854748] system 00:04: [mem 0xfec01000-0xfec01fff] could not be reserved
[    0.854750] system 00:04: [mem 0xfed80000-0xfed8ffff] could not be reserved

Finally lets show the configs of the VMs

## Working config for Windows Gaming VM with some comments

root@superpve:~# cat /etc/pve/qemu-server/101.conf
agent: 1
balloon: 0 #turn off ballooning highly recommended
bios: ovmf
boot: order=ide0;net0
cores: 16
cpu: host,hidden=1,flags=+aes #all 3 are important here
efidisk0: local:101/vm-101-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:2f:00,pcie=1 #AMD GPU all functions, pcie express and rom-bar are checked, Primary-GPU is not
ide0: local:101/vm-101-disk-1.qcow2,cache=none,discard=on,size=480G,ssd=1 #cache and discard are recommended for when underlying storage is ZFS like mine
machine: pc-q35-8.0
memory: 32768
meta: creation-qemu=8.0.2,ctime=1690042732
name: win11
net0: virtio=7E:21:63:3F:1F:21,bridge=vmbr0,firewall=1,mtu=9000
numa: 0 #5950x doesn't need NUMA maybe I'm wrong
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=571d782f-fd88-41b4-9fcf-9e726f25a2ed
sockets: 1
tpmstate0: local:101/vm-101-disk-2.raw,size=4M,version=v2.0
usb0: host=046d:c52b,usb3=1
usb1: host=1532:0266,usb3=1
usb2: host=1532:0c05,usb3=1
usb3: host=1532:0f13
usb4: host=1462:7c35
vga: none #No display though a VGA display option also works and 2 displays come up in VM
vmgenid: b3f09c2d-ad06-4395-bf6c-98ed5a43e5b4

## Similar Working config for MacOS VM config with some comments

root@superpve:~# cat /etc/pve/qemu-server/102.conf
## HSVE Proxmox Ventura Pre-Configured
## Credit goes to Harvey at https://hsve.org/download-links/
agent: 1
args: -device isa-applesmc,osk="<infamous apple key>" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off -cpu Haswell-noTSX,vendor=GenuineIntel,+hypervisor,kvm=on,vmware-cpuid-freq=on
balloon: 0 #turn off ballooning highly recommended
bios: ovmf
boot: order=ide2;virtio0;sata0
cores: 8
cpu: Haswell # don't know why but this works
efidisk0: local:102/vm-102-disk-0.qcow2,efitype=4m,size=528K
hostpci0: 0000:2f:00,pcie=1 # GPU select rombar,pcie and all functions but not Primary GPU it doesnt work for AMD gpus
machine: q35
memory: 32768
meta: creation-qemu=7.2.0,ctime=1685719709
name: macOS
net0: virtio=EE:DD:C8:04:95:10,bridge=vmbr0,mtu=9000
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=3600d55e-0334-4001-84e6-64ed97cc5bf1
sockets: 1
tags:
usb0: host=1532:0266,usb3=1
usb1: host=046d:085e,usb3=1
usb2: host=046d:c52b,usb3=1
vga: none #Same no display
virtio0: local:102/vm-102-disk-1.qcow2,cache=none,discard=on,iothread=1,size=250G
vmgenid: 0e432372-f00e-4ad9-9925-fc048c585dd4

lastly the plex ubuntu vm with nvidia passthrough is pretty normal looking config, nothing I do really seems to affect and pretty flexible on whatever config i give it

##Summary
So in summary I am still very much a noob, would love if some experienced ppl can explain why the AMD passthrough works without actually putting the pci-ids, also seems to be very sensitive on the cmdline parameters, no ballooning ram, no resize bar turned on in bios, and lastly not selecting primary gpu. All conditions together make this work.

Not sure if this was just known for AMD GPU passthrough but this was relatively hard to compile or it was my hardware with the dual gpu passthrough and zfs that made it somehow complex.

 

[dreamworks]

hi thanks for this, i tried my

MSI Radeon RX 6500 XT MECH 2X 4G OC Gaming Graphics Card - 4GB GDDR6, 2825 MHz, PCI Express 4 x 4, 64-bit, 1 x DP v 1.4a, HDMI 2.1​

but not working

 

[rupint]

hi thanks for this, i tried my

MSI Radeon RX 6500 XT MECH 2X 4G OC Gaming Graphics Card - 4GB GDDR6, 2825 MHz, PCI Express 4 x 4, 64-bit, 1 x DP v 1.4a, HDMI 2.1​

but not working

Above is very specific example when doing dual GPUs, you only have one GPU.

Follow the normal guide first to passthrough GPU, then you try the things like changing cmdline parameters. You can't copy paste from above is basically the answer.

Also you can check the syslog when you start a vm with gpu attached to debug as well as other debug commands I have above

 

[leesteken]

So in summary I am still very much a noob, would love if some experienced ppl can explain why the AMD passthrough works without actually putting the pci-ids, also seems to be very sensitive on the cmdline parameters, no ballooning ram, no resize bar turned on in bios, and lastly not selecting primary gpu. All conditions together make this work.

The 6800XT resets properly and you don't need to blacklist/early bind to vfio-pci to prevent Proxmox from touching it before the VM starts.
Resizable BAR is not supported by QEMU/Proxmox. The Primary GPU setting is a work-around for NVidia GPUs and not needed/wanted for AMD GPUs.
Ballooning does not work for VMs that use passthrough because the devices can do DMA at any time and the VM memory must therefore be locked in to host RAM.
I hope this explains a few things. More information is on the forum but sometimes hard to find.

Not sure if this was just known for AMD GPU passthrough but this was relatively hard to compile or it was my hardware with the dual gpu passthrough and zfs that made it somehow complex.

I don't know, I find AMD GPUs much easier (even with vendor-reset) but most guides out there are outdated, needlessly convoluted and geared towards NVidia GPUs.