QEMU 5 and Legacy IGD passthrough

[sshaikh]

6.2 may have broken VMs that use legacy IGD passthrough.

This is somewhat related to https://forum.proxmox.com/threads/igd-passthrough-setup-broke-with-proxmox-6-2.69670, except legacy IGD PT also relied on a quirk in qemu. This quirk used to be generally available, but since QEMU 5 has become optional at compile time (see this commit).

I'm therefore trying to build a new version of qemu-system-x86_64 that has the required config, but am getting a bit lost in the Proxmox source code. I've gotten as far as checking out https://git.proxmox.com/git/pve-qemu.git and updating all the submodules but am having trouble getting something to build.

I also need to know which config Proxmox is currently using, so any guidance would be appreciated.

 

[Stefan_R]

This is somewhat related to https://forum.proxmox.com/threads/igd-passthrough-setup-broke-with-proxmox-6-2.69670, except legacy IGD PT also relied on a quirk in qemu. This quirk used to be generally available, but since QEMU 5 has become optional at compile time (see this commit).

Can't test this to say for certain right now, but from how I read the commit the config should be enabled by default if PCI support is enabled (which it definitely is). There isn't even an option in the configure file to set or unset this. This might be an upstream bug then, maybe building a version with the patch reverted might make sense for testing.

I'm therefore trying to build a new version of qemu-system-x86_64 that has the required config, but am getting a bit lost in the Proxmox source code. I've gotten as far as checking out https://git.proxmox.com/git/pve-qemu.git and updating all the submodules but am having trouble getting something to build.

Check out our developer docs for general build instructions. There shouldn't be anything QEMU specific, other than 'make deb' has to be run as root. For easiest workflow I'd recommend compiling on a Debian or even directly on a PVE installation.

I also need to know which config Proxmox is currently using, so any guidance would be appreciated.

See debian/rules in pve-qemu.

 

[sshaikh]

Thanks for the reply. I've since made some progress but am now getting the following error trying to compile in a debian 10 container. I'll read the dev docs to see if there are any clues.

In file included from /root/pve-qemu/pve-qemu-kvm-5.0.0/include/qemu/timer.h:4,
                 from /root/pve-qemu/pve-qemu-kvm-5.0.0/include/qemu/timed-average.h:29,
                 from /root/pve-qemu/pve-qemu-kvm-5.0.0/include/block/accounting.h:28,
                 from /root/pve-qemu/pve-qemu-kvm-5.0.0/include/block/block_int.h:27,
                 from block/file-posix.c:30:
/usr/include/linux/swab.h: In function ‘__swab’:
/root/pve-qemu/pve-qemu-kvm-5.0.0/include/qemu/bitops.h:20:34: error: "sizeof" is not defined, evaluates to 0 [-Werror=undef]
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
                                  ^~~~~~
/root/pve-qemu/pve-qemu-kvm-5.0.0/include/qemu/bitops.h:20:41: error: missing binary operator before token "("
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)

 

[Stefan_R]

Ah, yeah, that's a known issue with the libc library actually. Try installing the package 'pve-kernel-libc-dev' from pve-no-subscription, that should fix that issue.

 

[sshaikh]

Ah, thanks for the insight, that package did the trick.

Regarding the QEMU5 commit I understood the same, that it should be there with CONFIG_PC_PCI, so reverting the patch will be part of my investigation.

See debian/rules in pve-qemu.

I'm specifically looking for where for example CONFIG_PC_PCI might be defined. I'm still digging, but it would be a mak file somewhere.

 

[Stefan_R]

I'm specifically looking for where for example CONFIG_PC_PCI might be defined. I'm still digging, but it would be a mak file somewhere.

default-configs/i386-softmmu.mak sets CONFIG_i440FX and CONFIG_Q35 which in turn sets CONFIG_PC_PCI via hw/i386/Kconfig

 

[sshaikh]

So I've verified that the normal build pulls in the refactored IGD code, and that it still doesn't work.

Reverting the patch however does.

I'll follow up with upstream, but in the meantime would the pve patches be doing something? I do doubt it but I thought I'd ask.

 

[Stefan_R]

So I've verified that the normal build pulls in the refactored IGD code, and that it still doesn't work.

Reverting the patch however does.

That's progress!

I'll follow up with upstream, but in the meantime would the pve patches be doing something? I do doubt it but I thought I'd ask.

If they push a follow-up fix let me know, I'm sure we could pick it up for pve-qemu if it's straightforward enough.

Our patches don't change anything regarding IGD AFAIK, but QEMU is a mighty beast so take that with a grain of salt. You could probably try this out by building only with the minimum required patches and testing with that (usually 0019-PVE-Add-dummy-id-command-line-parameter.patch alone is enough to make qemu-server happy).

 

[sshaikh]

Instead of messing around with Proxmox patches I've reproduced the bug (and fix) in clean upstream sources. Filed a bug here:

https://bugs.launchpad.net/qemu/+bug/1882784

Thanks for the attention and help - I'm not sure if this should be marked solved as of now?

I'll be using a patched QEMU for this - should I just drop it in /usr/bin?

 

[Stefan_R]

Instead of messing around with Proxmox patches I've reproduced the bug (and fix) in clean upstream sources. Filed a bug here:

https://bugs.launchpad.net/qemu/+bug/1882784

It's usually more effective to write directly to the qemu-devel mailing list, you could even submit the fix you mention in your report directly. If it gets accepted upstream we could definitely carry it, if that's really all there is to it.

Thanks for the attention and help - I'm not sure if this should be marked solved as of now?

However you want, I personally wouldn't since the matter is still open.

I'll be using a patched QEMU for this - should I just drop it in /usr/bin?

Either that or you can add it as a .patch file to debian/patches/pve/ and add it to debian/patches/series so you can build yourself a .deb package to install.

 

[sshaikh]

My fix is simply defining CONFIG_VFIO_IGD which I'm not sure will wash with them . I'm not familiar enough with their process to see how to introduce a preprocessor variable from a kconfig variable.

But since Proxmox always has CONFIG_PCI I might make it a deb patch. If I do, and install, will it interfere with future Proxmox updates via apt?

EDIT: actually scrap that. The aarch64 target fails if I build it with IGD enabled, so it has to be driven by the kconfig. I'll figure something more general out eventually but for now I'll just build a patched qemu-system-x86_64.

 

[Stefan_R]

But since Proxmox always has CONFIG_PCI I might make it a deb patch. If I do, and install, will it interfere with future Proxmox updates via apt?

it won't interfere, but updates on our side will override yours on upgrade. You can avoid it by setting "pve-qemu-kvm" on hold for example.

 

[sshaikh]

Okay. Great - it being overridden is fine. In the meantime I figured out a patch that will work for all builds:

diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
index 2d348f8237..a633df965e 100644
--- a/hw/vfio/pci-quirks.c
+++ b/hw/vfio/pci-quirks.c
@@ -25,6 +25,7 @@
 #include "hw/qdev-properties.h"
 #include "pci.h"
 #include "trace.h"
+#include "config-devices.h"

 /*
  * List of device ids/vendor ids for which to disable

The same fix was simultaneously offered by QEMU on the bug so it looks like it should be patched soon. I'm now interested in the deb build process so I'll see if I can get a deb patch working anyway.

 

[sshaikh]

Just to report back that a deb-patched deb package has brought the functionality back via Proxmox server (subject to suppressing PCI bridge 2 as per https://forum.proxmox.com/threads/igd-passthrough-and-hard-coded-pci-bridges.68285)

 

[sshaikh]

FYI the official patch has now been included:

https://bugs.launchpad.net/qemu/+bug/1882784

 

[tachang]

What would be the best way to apply this patch? Wait for a Proxmox dot release or build QEMU from source?

 

[Stefan_R]

The full fixes will be included in qemu-server 6.2-4 and pve-qemu-kvm 5.0.0-5, which should hit the pve-no-subscription repository soon. You can build the packages from source as well, if you don't want to wait (see https://git.proxmox.com).

With those packages installed, you can specify "legacy-igd=1" in your <vmid>.conf for the hostpci device to enable the new mode.

 

[sshaikh]

If you already know how to build things then you'll have this fixed in less than 15 minutes.

If you don't then it's a fun (and safe enough) exercise to do that will take 2 hours or so.

 

[tachang]

Which git repo is pve-qemu-kvm ? It says there is an older one but it moved but no indication where it moved to?

 

[sshaikh]

The repo is pve-qemu