When starting VM with the iGPU the whole system crashes (AMD iGPU Passtrough)

[Nic_]

PVE: 8.0.4

(from neofetch)
CPU: AMD Ryzen 9 7950X3D (32) @ 4.200GHz
GPU: AMD ATI 0b:00.0 Raphael

What I did:
1. Added quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction to /etc/default/grub
2. Added these modules to /etc/modules

 vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd

3. Added these lines into a .conf file in /etc/modprobe.d:

blacklist amdgpu
blacklist radeon
options vfio_iommu_type1 allow_unsafe_interrupts=1
options vfio-pci ids=0000:0b:00.0

4. Added this PCI Device config to the VM:


This is the output for the iGPU when I do lspci -nnk:

0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c9)
        Subsystem: ASUSTeK Computer Inc. Raphael [1043:8877]
        Kernel modules: amdgpu

The VM is a Windows 10 VM and when I start it whole PVE crashes
If you need logs just tell me please

Thanks

 

[Dunuin]

Whats the output of pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""? When your iGPU is sharing a IOMMU group with other devices and you start that VM all other devices will be removed from the host and passed through as well. If things like SATA controllers are in there it will then of cause crash the host.

 

[Nic_]

Whats the output of pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""? When your iGPU is sharing a IOMMU group with other devices and you start that VM all other devices will be removed from the host and passed through as well. If things like SATA controllers are in there it will then of cause crash the host.

No device shares the IOMMU group with the iGPU.

 

[tabish]

I experienced the same thing today.

Added this group to a Windows VM, which is basically a IOMMU group of AMD integrated GPU of AMD 7900X3D. Now i don't really know what I actually did there. But as soon as I turned on the VM, the whole proxmox server crashed.

 

[Nic_]

I experienced the same thing today.
View attachment 67886
Added this group to a Windows VM, which is basically a IOMMU group of AMD integrated GPU of AMD 7900X3D. Now i don't really know what I actually did there. But as soon as I turned on the VM, the whole proxmox server crashed.

If you only have the iGPU in your server, proxmox also uses this GPU to display the terminal when you attach a monitor to your server. Now if you allocate that iGPU to your VM, it interferes with the proxmox server itself and therefore it crashes.

Thats what I believe is happening. I can't 100% confirm this as I only read that in some other forum thread, but it makes sense to me.
I just bought a cheap dGPU and put it into my server and everything worked fine that way

 

[wrobelda]

If you only have the iGPU in your server, proxmox also uses this GPU to display the terminal when you attach a monitor to your server. Now if you allocate that iGPU to your VM, it interferes with the proxmox server itself and therefore it crashes.

No, that's not how things work on Linux. The crash is down to hardware/BIOS/UEFI, not Linux crashing because it needs a console to display. Linux console is just a tty terminal, it's not integral part of the system. Besides, you can blacklist amdgpu and have Linux basically be completely unaware of the GPU's presence.

For the record, I am having same exact issue with 7900 and Asus B650 Creator motherboard.
EDIT: sorted it out, I had "All functions" enabled on GPU device, which resulted in Proxmox passing-through a complete PCI controller root node.

 

[Nic_]

No, that's not how things work on Linux. The crash is down to hardware/BIOS/UEFI, not Linux crashing because it needs a console to display. Linux console is just a tty terminal, it's not integral part of the system. Besides, you can blacklist amdgpu and have Linux basically be completely unaware of the GPU's presence.

For the record, I am having same exact issue with 7900 and Asus B650 Creator motherboard.
EDIT: sorted it out, I had "All functions" enabled on GPU device, which resulted in Proxmox passing-through a complete PCI controller root node.

Ohh thanks for explaining