Hi this is more documentation for me and to provide help to fellow forum lurkers who might be stuck on the same configs.
## Intro
So following the normal guides to passthrough GPUs got the Nvidia GPU working fine in a plex ubuntu vm but the AMD would not work for a while, finally figured it out over this weekend.
## Flex
What I have in my machine (and what I do with it):
Proxmox 8.0.3 bookworm
AMD Ryzen 9 5950X (amd resizable bar and 4g decoding are both disabled in Bios necessary for VM to boot and also not supported inside vm so anyways a waste)
128 GB RAM
AMD 6800XT (Passthrough to Windows Gaming VM or MacOS Ventura VM can only boot one)
Nvidia A2000 (Passthrough to Plex Ubuntu)
10gig Networking connected to a NAS box with dump backups
3 Pcie4.0 SSDs of 2TB capacity each (native Raidz1 ZFS for Proxmox)
6 SATA ssds - 8TB(3)/4TB(3) (Passthrough to Truenas Scale) with a zpool made of z1+z1 vdevs
That was my flex
on with the rest
Since I use ZFS in proxmox my cmdline is not in
You must use
My config files mostly the
Use
Debug commands that helped me:
Another one
Finally lets show the configs of the VMs
## Working config for Windows Gaming VM with some comments
## Similar Working config for MacOS VM config with some comments
lastly the plex ubuntu vm with nvidia passthrough is pretty normal looking config, nothing I do really seems to affect and pretty flexible on whatever config i give it
##Summary
So in summary I am still very much a noob, would love if some experienced ppl can explain why the AMD passthrough works without actually putting the pci-ids, also seems to be very sensitive on the cmdline parameters, no ballooning ram, no resize bar turned on in bios, and lastly not selecting primary gpu. All conditions together make this work.
Not sure if this was just known for AMD GPU passthrough but this was relatively hard to compile or it was my hardware with the dual gpu passthrough and zfs that made it somehow complex.
## Intro
So following the normal guides to passthrough GPUs got the Nvidia GPU working fine in a plex ubuntu vm but the AMD would not work for a while, finally figured it out over this weekend.
## Flex
What I have in my machine (and what I do with it):
Proxmox 8.0.3 bookworm
AMD Ryzen 9 5950X (amd resizable bar and 4g decoding are both disabled in Bios necessary for VM to boot and also not supported inside vm so anyways a waste)
128 GB RAM
AMD 6800XT (Passthrough to Windows Gaming VM or MacOS Ventura VM can only boot one)
Nvidia A2000 (Passthrough to Plex Ubuntu)
10gig Networking connected to a NAS box with dump backups
3 Pcie4.0 SSDs of 2TB capacity each (native Raidz1 ZFS for Proxmox)
6 SATA ssds - 8TB(3)/4TB(3) (Passthrough to Truenas Scale) with a zpool made of z1+z1 vdevs
That was my flex
Since I use ZFS in proxmox my cmdline is not in
/etc/default/grub
but in /etc/kernel/cmdline
omit the starting part root=ZFS=rpool/ROOT/pve-1 boot=zfs
when using grub. My /etc/kernel/cmdline
:
Bash:
root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt textonly quiet vfio_iommu_type1 allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction nofb nomodeset initcall_blacklist=sysfb_init
You must use
initcall_blacklist=sysfb_init
instead of video=efifb:off
the later no longer works above 5.15 kernel (Proxmox 7.x) I believeMy config files mostly the
/etc/modprobe.d/vfio.conf
is where the magic happens:
Bash:
root@superpve:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
### donot blacklist amdgpu
root@superpve:~# cat /etc/modprobe.d/kvm.conf
cat: /etc/modprobe.d/kvm.conf: No such file or directory
### I deleted this file and moved the ignore_msrs line into vfio.conf the next file
root@superpve:~# cat /etc/modprobe.d/vfio.conf
options kvm ignore_msrs=1 ### moved from kvm.conf
options vfio-pci ids=10de:2531,10de:228e disable_vga=1 ### this is only the pci ids of the Nvidia the AMD ones are not included not sure what black magic this is
softdep amdgpu pre: vfio vfio_pci ### dont know what this means but doesnt work without it and basically I think forces the AMD gpu to use vfio driver
root@superpve:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
### as expected from generic passthrough guide
root@superpve:~# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
### as expected from generic passthrough guide
Use
update-initramfs -k all -u && update-grub && pve-efiboot-tool refresh
to refresh your boot configDebug commands that helped me:
Bash:
root@superpve:~# cat /proc/iomem | grep -e BOOTFB
root@superpve:~# should be no output
root@superpve:~# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.105072] AMD-Vi: Unknown option - 'on'
[ 0.249957] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[ 0.864314] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.866486] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.866486] AMD-Vi: Extended features (0x58f77ef22294a5a, 0x0): PPR NX GT IA PC GA_vAPIC
[ 0.866489] AMD-Vi: Interrupt remapping enabled
[ 0.866813] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 4.280988] AMD-Vi: AMD IOMMUv2 loaded and initialized
root@superpve:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done;
root@superpve:~# above verifies everything has different iommu groups
Another one
dmesg | grep -e "can't reserve [mem"
but mine was this could not be reserved as shown below. Not sure why I have this error but everything seems to still be working.
Bash:
root@superpve:~# dmesg | grep -e "could not be reserved"
[ 0.854747] system 00:04: [mem 0xfec00000-0xfec00fff] could not be reserved
[ 0.854748] system 00:04: [mem 0xfec01000-0xfec01fff] could not be reserved
[ 0.854750] system 00:04: [mem 0xfed80000-0xfed8ffff] could not be reserved
Finally lets show the configs of the VMs
## Working config for Windows Gaming VM with some comments
Bash:
root@superpve:~# cat /etc/pve/qemu-server/101.conf
agent: 1
balloon: 0 #turn off ballooning highly recommended
bios: ovmf
boot: order=ide0;net0
cores: 16
cpu: host,hidden=1,flags=+aes #all 3 are important here
efidisk0: local:101/vm-101-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:2f:00,pcie=1 #AMD GPU all functions, pcie express and rom-bar are checked, Primary-GPU is not
ide0: local:101/vm-101-disk-1.qcow2,cache=none,discard=on,size=480G,ssd=1 #cache and discard are recommended for when underlying storage is ZFS like mine
machine: pc-q35-8.0
memory: 32768
meta: creation-qemu=8.0.2,ctime=1690042732
name: win11
net0: virtio=7E:21:63:3F:1F:21,bridge=vmbr0,firewall=1,mtu=9000
numa: 0 #5950x doesn't need NUMA maybe I'm wrong
ostype: win11
scsihw: virtio-scsi-single
smbios1: uuid=571d782f-fd88-41b4-9fcf-9e726f25a2ed
sockets: 1
tpmstate0: local:101/vm-101-disk-2.raw,size=4M,version=v2.0
usb0: host=046d:c52b,usb3=1
usb1: host=1532:0266,usb3=1
usb2: host=1532:0c05,usb3=1
usb3: host=1532:0f13
usb4: host=1462:7c35
vga: none #No display though a VGA display option also works and 2 displays come up in VM
vmgenid: b3f09c2d-ad06-4395-bf6c-98ed5a43e5b4
## Similar Working config for MacOS VM config with some comments
Bash:
root@superpve:~# cat /etc/pve/qemu-server/102.conf
## HSVE Proxmox Ventura Pre-Configured
## Credit goes to Harvey at https://hsve.org/download-links/
agent: 1
args: -device isa-applesmc,osk="<infamous apple key>" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off -cpu Haswell-noTSX,vendor=GenuineIntel,+hypervisor,kvm=on,vmware-cpuid-freq=on
balloon: 0 #turn off ballooning highly recommended
bios: ovmf
boot: order=ide2;virtio0;sata0
cores: 8
cpu: Haswell # don't know why but this works
efidisk0: local:102/vm-102-disk-0.qcow2,efitype=4m,size=528K
hostpci0: 0000:2f:00,pcie=1 # GPU select rombar,pcie and all functions but not Primary GPU it doesnt work for AMD gpus
machine: q35
memory: 32768
meta: creation-qemu=7.2.0,ctime=1685719709
name: macOS
net0: virtio=EE:DD:C8:04:95:10,bridge=vmbr0,mtu=9000
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=3600d55e-0334-4001-84e6-64ed97cc5bf1
sockets: 1
tags:
usb0: host=1532:0266,usb3=1
usb1: host=046d:085e,usb3=1
usb2: host=046d:c52b,usb3=1
vga: none #Same no display
virtio0: local:102/vm-102-disk-1.qcow2,cache=none,discard=on,iothread=1,size=250G
vmgenid: 0e432372-f00e-4ad9-9925-fc048c585dd4
lastly the plex ubuntu vm with nvidia passthrough is pretty normal looking config, nothing I do really seems to affect and pretty flexible on whatever config i give it
##Summary
So in summary I am still very much a noob, would love if some experienced ppl can explain why the AMD passthrough works without actually putting the pci-ids, also seems to be very sensitive on the cmdline parameters, no ballooning ram, no resize bar turned on in bios, and lastly not selecting primary gpu. All conditions together make this work.
Not sure if this was just known for AMD GPU passthrough but this was relatively hard to compile or it was my hardware with the dual gpu passthrough and zfs that made it somehow complex.
Last edited: