Can't reboot VM with PCIe Passthrough without freezing Proxmox node

[Enigma]

I was able to install the AMD drivers without crashing but upon rebooting the VM, the entire Proxmox node froze up and had to be manually reset. I figured the VM was borked but actually if I start it up and shut it down without using the 'Restart' mechanism, it works fine.

lspci

PCIe Slot 1: (Proxmox Server)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos XTX [Radeon HD 8490 / R5 235X OEM]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]

PCIe Slot 3: (Windows 10 VM)
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 2100]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aae0

cat /etc/pve/qemu-server/104.conf

agent: 1
bios: ovmf
boot: cd
bootdisk: scsi1
cores: 8
efidisk0: SSD1:vm-104-disk-2,size=128K
ide0: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
memory: 4096
name: Win10x64
net0: virtio=00:0C:29:06:F2:63,bridge=vmbr0
numa: 0
ostype: win10
scsi1: SSD1:vm-104-disk-1,size=90G
scsihw: virtio-scsi-pci
smbios1: uuid=e21f79cd-6b3a-4da0-841c-0892b92ea3b8
sockets: 1
machine: q35
hostpci0: 06:00,pcie=1,x-vga=on

cat vfio.conf
options vfio-pci ids=1002:6995,1002:aae0 disable_vga=1

lspci -n -s 01:00

01:00.0 0300: 1002:6771
01:00.1 0403: 1002:aa98

lspci -n -s 06:00

06:00.0 0300: 1002:6995
06:00.1 0403: 1002:aae0

cat blacklist.conf 

#blacklist radeon //Proxmox host is still using a radeon GPU AFAIK
blacklist nouveau
blacklist nvidia

find /sys/kernel/iommu_groups/ -type l | grep 18

/sys/kernel/iommu_groups/18/devices/0000:01:00.1
/sys/kernel/iommu_groups/18/devices/0000:01:00.0

find /sys/kernel/iommu_groups/ -type l | grep 22

/sys/kernel/iommu_groups/22/devices/0000:06:00.0
/sys/kernel/iommu_groups/22/devices/0000:06:00.1

dmesg | grep -e DMAR -e IOMMU

[    1.344032] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    8.728815] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    8.728816] AMD IOMMUv2 functionality not available on this system

cat /etc/modules
...
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Motherboard: ASUS Sabertooth 990FX r2.0
CPU: FX-8370

So far, GPU performance seems to also be quite low compared to bare metal. I wasn't able to launch Unigine Valley due to missing MSVC...DLL's.

I could however connect to the system via Parsec which requires hardware encoding to function. The performance again wasn't what bare metal would provide though.

 

[Enigma]

Anyone have any ideas on how to fix this or what might be wrong with the configuration?

 

[Enigma]

Proxmox system seems to crash less with `x-vga=on` removed from the `hostpci0` line.

The actual VM however crashes when trying to launch a game so something is definitely wrong.

I have amdgpu blacklisted and am guessing this doesn't matter or is fine:

lspci -k

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 2100]
   Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b0c
   Kernel driver in use: vfio-pci
   Kernel modules: amdgpu

 

[Enigma]

Still having this issue... Maybe something about disabling NPT on the CPU might help?

 

[ONE FOTON]

I have amdgpu blacklisted and am guessing this doesn't matter or is fine:

I also put it on the blacklist. and x-vga = on also delete it from vm.conf
the vm recognizes the gpu (rx480), but I installed the drivers amdgpu pro and the drivers do not see it ... I do not know what happens!
I think that the modules are not loaded to the kernel, but I have no idea the name of the modules that I have to load.

 

[Enigma]

I also put it on the blacklist. and x-vga = on also delete it from vm.conf
the vm recognizes the gpu (rx480), but I installed the drivers amdgpu pro and the drivers do not see it ... I do not know what happens!
I think that the modules are not loaded to the kernel, but I have no idea the name of the modules that I have to load.

`lspci -k` should tell you.

 

[ONE FOTON]

`lspci -k` should tell you.

Thanks for the help!
I keep the same error! and no info comes out in the proxmox kernel syslog!
Will you know where the virtual machin logs are?
right now after installing the amdgpu pro drivers to vm I can not boot with pcie enabled.
the host network is broken and I have to reboot everything!
I will try to disable pcie in the guest, configure the guest to load the amdgpu modules and hope it works!

 

[Enigma]

I'm not sure myself. PCIe passthrough is still not properly working for me. I can get low level hardware acceleration to work (12FPS or so) but any 3d application seems to crash the system as mentioned above.

If your network is cutting out, you might have a shared iommu_group and you're pretty much out of luck with that PCIe slot. If your Proxmox system is straight up seizing like mine though then yeah I'm in a similar boat - not sure.

I'm able to install the drivers in Windows 10 for the GPU but that's about it. If you are using a Linux guest, I think you might be better off not installing amdgpu pro drivers (the appropriate drivers should be installed by default via mesa)

 

[ONE FOTON]

yes, yes, yes !!!
I added the modules to etc/rc.local and removed some that had to do with the pcie and now boot!
and more or less I'm still not satisfied.
I think that's what worked! although I also untie the groups like menciana here (https://wiki.debian.org/VGAPassthrough)
and the passthrough I did it this way
hostpci0: xx:xx,x-vga=on<<<<<<xx go your vendor and id pcie
Do you know where the files are for modifying the gpu bios?

 

[Enigma]

Sorry you lost me. Hopefully someone else from @proxmox can comment.

 

[Enigma]

Still having no luck with this.