[SOLVED] GPU Passthrough Issues After Upgrade to 7.2

Thanks

After I upgrade to 5.15.39-1 kernel, I still cannot solve the passthrough GPU issue. The config worked in 7.1

kvm: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=1,romfile=/root/vbios.rom: Failed to mmap 0000:00:02.0 BAR 2. Performance may be slow
kvm: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=1,romfile=/root/vbios.rom: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0
TASK OK
Have you tried this work-around?
 
after adding "initcall_blacklist=sysfb_init" to kernel parameter, it work again.

however, still have this error

kvm: -device vfio-pci,host=00:02.0,addr=0x02,x-igd-gms=1,romfile=vbios.bin: IGD device 0000:00:02.0 cannot support legacy mode due to existing devices at address 1f.0

Thanks
 
Last edited:
It's already autumn, and the issue has not yet been resolved.
My video cards still do not want to work through the crutch method with scripts.
 
PBS runs this gpu-hookscript so it kills the other vm that's using it
Code:
INFO: creating Proxmox Backup Server archive 'vm/101/2022-09-22T14:00:00Z'
INFO: starting kvm to execute backup task
gpu-hookscript: Resetting GPU for Vitual Machine 101
ERROR: Backup of VM 101 failed - PCI device '0000:0f:00.0' already in use by VMID '112'
INFO: Failed at 2022-09-23 00:00:03
 
Depends on hardware, this patch don't work on many. Best will be to wait for the 7.2-2 hopefully by december. But for a fully working prox: disconnect modem, install prox 7.1-2 , set the usual 8-10 setting and hop you go. Then you can check if you realy in an urge to do update.. but as a server is not a phone with update every 30min...
 
With 7.2 (and kernel 5.15 on 7.1), I would expect video=simplefb:off to fix this (where video=efifb:off worked in 5.13), but I my experience this does not work for every GPU and BOOTFB does not release memory (as can be seen in /proc/iomem).
I found that, for AMD GPUs, un-blacklisting amdgpu and not early binding to vfio_pci and removing video=efifb:off and similar parameters works best. amdgpu just takes over from the BOOTFB, and does release the GPU nicely to vfio_pci when starting the VM. (Of course, for AMD vendor-reset and reset_method=device_specific might be required.)
I don't know if this also works for nouveau or i915.
Works To me
 
This recently stopped working for me (gpu-hookscript.sh). Originally I followed the suggestion not to blacklist the amdgpu which worked.
Since then sometime during multiple kernel upgrade and an upgrade to windows 11 or some windows 11 update, it stopped working and the host would crash without any messages immediately when I shutdown or rebooted the win 11 vm.

This seems to be fixed after reblacklisting the amdgpu. Now I have this:
/etc/modprobe.d/pve-blacklist.conf
blacklist snd_hda_intel
blacklist amdgpu
blacklist radeon
 
This recently stopped working for me (gpu-hookscript.sh). Originally I followed the suggestion not to blacklist the amdgpu which worked.
Since then sometime during multiple kernel upgrade and an upgrade to windows 11 or some windows 11 update, it stopped working and the host would crash without any messages immediately when I shutdown or rebooted the win 11 vm.
I haven't used the recent 5.15 kernels as switching to kernel 5.19 made amdgpu much better in the hand-off to vfio-pci. Once again, I can use the AMD GPU to boot (to see the boot messages), pass it to a VM and afterwards give it back to the Proxmxo host (to see the shut-down messages).
 
  • Like
Reactions: janssensm
I haven't used the recent 5.15 kernels as switching to kernel 5.19 made amdgpu much better in the hand-off to vfio-pci. Once again, I can use the AMD GPU to boot (to see the boot messages), pass it to a VM and afterwards give it back to the Proxmxo host (to see the shut-down messages).
is the gpu-hookscript still needed with the 5.19 kernel? what's in you cmdline and modprobe.d vfio-pci settings?
 
It depends what you do in "the" hookscript. For certain AMD GPUs, you do always need vendor-reset.
My gpu is amd but I don't use the vendor reset. I use the remove; rescan method suggested very early in this thread.
Do you explicitly reserve the device for vfio-pci with kernel options or module options or is that no longer required in k5.19?
 
Last edited:
My gpu is amd but I don't use the vendor reset. I use the remove; rescan method suggested very early in this thread.
Do you explicitly reserve the device for vfio-pci with kernel options or module options or is that no longer required in k5.19?
No, I just let amdgpu load for the GPU to display boot messages, but that's also why I need vendor-reset. I use a hookscript to select device_specific reset and to rebind the GPU to amdgpu during host shutdown to see the shutdown messages.
 
  • Like
Reactions: janssensm
No, I just let amdgpu load for the GPU to display boot messages, but that's also why I need vendor-reset. I use a hookscript to select device_specific reset and to rebind the GPU to amdgpu during host shutdown to see the shutdown messages.
That's terrific but a little low on detail.
How do I determine the "vendor specific" reset for my gpu is?
Could you post your hookscript here?
 
That's terrific but a little low on detail.
How do I determine the "vendor specific" reset for my gpu is?
Is your GPU supported by vendor-reset? Then you need to install and load vendor-reset. And do you run kernel 5.15 or higher? Then you need to do this also (that is not a bug, it's how the new system works).
Could you post your hookscript here?
Bash:
#!/bin/bash
GPU='0000:0b:00.0'
[ "$2" == "pre-start" ] && echo device_specific >"/sys/bus/pci/devices/${GPU}/reset_method"
 
Last edited:
  • Like
Reactions: Kodey
I haven't used the recent 5.15 kernels as switching to kernel 5.19 made amdgpu much better in the hand-off to vfio-pci. Once again, I can use the AMD GPU to boot (to see the boot messages), pass it to a VM and afterwards give it back to the Proxmxo host (to see the shut-down messages).
No, I just let amdgpu load for the GPU to display boot messages, but that's also why I need vendor-reset. I use a hookscript to select device_specific reset and to rebind the GPU to amdgpu during host shutdown to see the shutdown messages.
Thanks so much for all your earlier info regarding gpu passthrough.
I notice indeed a nice handoff from amdgpu to vfio, running kernel 5.19. So at the moment no blacklisting and nothing early loading in vfio. Just let amdgpu boot and run on the host. When starting the vm, vfio takes over the card.
The only thing I wanted is the vm handing off the card from vfio and returning it to amdgpu when shutting down the vm. No card reset necessary, just unbind from vfio and rebind to amdgpu.
Configured that in a hookscript with in post-stop phase. Seems to work great.
 
  • Like
Reactions: leesteken
So just installed a fresh copy of Proxmox 7.3. Not sure if it is plagued with the same issues but I went through the PCI passthrough guide along with info provided by @nick.kopas . Not sure if it's still relevant with kernel 5.15.74. But I used it just in case.

I came up with the following Grub entry:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init"

Works perfectly with NVIDIA 3060ti on Ubuntu 22.04 with nvidia-drivers-520-open installed.
 
Hello!
Before the upgrade Proxmox, I had two video cards working: UHD 600 (J4105) and GT730.
After upgrade VM with UHD 600 throws an error on startup:
Code:
kvm: -device vfio-pci,host=0000:00:02.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:00:02.0: error getting device from group 1: Invalid argument
Verify all devices in group 1 are bound to vfio-<bus> or pci-stub and not already in use
TASK ERROR: start failed: QEMU exited with code 1
Please help!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!