VMs don't release passed-through GPU?

[stooovie]

Hi, my goal is to have an option to boot into Ubuntu and Win10 VM with Intel iGPU passthrough. That works.

What doesn't work is to be able to shut down the VM and have another VM or LXC (Jellyfin) use the same GPU. In order to do that, I have to fully reboot the host.

Goal is NOT to run two things using passthrough GPU at once, only to be able to shut down one and start another, without having to reboot the host. What can I do to do that? Thanks!

 

[leesteken]

If you want to pass the GPU from a VM to a container, then you need to unbind it from vfio-pci and load the proper driver for it (and make sure not to blacklist that driver).
If you have trouble passing the GPU from one VM to another VM, check whether it works after stopping (shutdown) and starting the same VM. If not, then the GPU does not reset properly and there is probably not much you can do to fix it.

 

[stooovie]

I can get Ubuntu with GPU passthrough to start, shutdown, and start up again without errors. I can not, however, get acceleration working in Jellyfish LXC after a VM with passthrough is shut down. It looks like I just have to reboot the host to get acceleration in the LXC working again.

So, do I somehow unbind the iGPU from vfio-pci? How? Can I automate it? Thank you!

 

[leesteken]

So, do I somehow unbind the iGPU from vfio-pci? How? Can I automate it? Thank you!

Yes. Here is an example for an AMD GPU which I hope can get you started. You can find out which PCI ID to use and can determine which driver is necessary for your iGPU with lspci -k. You can use a hookscript to automate it.

 

[stooovie]

It's interesting that I do not need to unbind anything when shutting down VMs and starting another (Win 10 and Ubuntu, both with the passthrough) - both start and work just fine. It's just the LXC that gives me issues.

Thanks for the tip!

 

[leesteken]

It's interesting that I do not need to unbind anything when shutting down VMs and starting another (Win 10 and Ubuntu, both with the passthrough) - both start and work just fine. It's just the LXC that gives me issues.

All VMs with passthrough all use the vfio-pci driver for VFIO passthrough (for any PCIe device). Containers use the actual device drivers via the host kernel.
Proxmox knows how to switch from any driver to vfio-pci (when the VM uses passthrough) but does not do the reverse (which also does not always work and sometimes there are more than one possible driver).

 

[stooovie]

Ha! unbinding worked! For posterity:

to unbind an Intel HD Graphics 620 (typically ID 00:02.0), I did

echo "0000:00:02.0" > "/sys/bus/pci/devices/0000:00:02.0/driver/unbind"

to bind it back to the device driver, I did

echo "0000:00:02.0" > /sys/bus/pci/drivers/i915/bind

both in host shell. After restarting the Jellyfin LXC, acceleration is working. No need to reboot the entire node host.
Thanks @leesteken! Now just to find out how to automate all of this

 

[stooovie]

@leesteken, could you please help me with this?

Proxmox 7.4-3, Ubuntu 20.04 SeaBIOS/i440fx (same with OMVF/q35) guest. iGPU (Intel 620 with i5-7200u) passed through.

I do see the output on my 4k Samsung TV over HDMI, but the resolution is limited at 1280x800 (or less), and none of the resolutions is 16x9 - either 4x3 or 16x10. I don't plan to game on it, but I would like to use it for emulation, and need proper aspect ratio. 3D acceleration seems to be working, but the output looks kinda weird, with lots of tearing. the iGPU is detected as "Mesa Intel HD Graphics 620 (KBL GT2)". Same on a Full HD display.

One more data point - the boot screen (spinner and Ubuntu logo) are fine - proper aspect ratio and seemingly full 4k resolution. Trouble starts at desktop.

The hardware is capable of running 4k/60Hz in Windows VM, so it's not a HW or HDMI issue. Any clues? Thanks!

TL;DR any Linux starts with native resolution but resets to 1280x800 (or less) on ANY resolution change.

EDIT: tried xrandr --newmode but to no avail. The new resolution IS added to the display settings but immediately revert back to 1280x800.

 

[leesteken]

@leesteken, could you please help me with this?

Sorry, I have no idea (and no experience with that harwdare what so ever).