IGD passthrough and hard coded PCI bridges

[sshaikh]

I have successfully managed to pass through my P4600 IGD to a VM. However I could only get it working up to a machine of pc-i440fx-2.2.

The error given with pc-i440fx > 2.2 was that the address 1f.0 was in use by another device and the vfio legacy IGD passthrough requires this address to be free. This was confirmed by trying to create an ISA bridge (vfio-pci-igd-lpc-bridge) at 1f.0 - we find that there is already a PCI bridge there.

Looking at the code at https://github.com/proxmox/qemu-server/blob/master/PVE/QemuServer.pm we see the folllowing:

    if (!$q35) {
    # add pci bridges
        if (qemu_machine_feature_enabled ($machine_type, $kvmver, 2, 3)) {
       $bridges->{1} = 1;
       $bridges->{2} = 1;
    }

    $bridges->{3} = 1 if $scsihw =~ m/^virtio-scsi-single/;

    while (my ($k, $v) = each %$bridges) {
        $pciaddr = print_pci_addr("pci.$k");
        unshift @$devices, '-device', "pci-bridge,id=pci.$k,chassis_nr=$k$pciaddr" if $k > 0;
    }
    }

Where qemu_machine_feature_enabled effectively checks to see what version of pc-i440fx we are using, and:

sub print_pci_addr {
    my ($id, $bridges) = @_;

    my $res = '';
    my $devices = {

#...

    'pci.1' => { bus => 0, addr => 30 },
    'pci.2' => { bus => 0, addr => 31 },

#...

    };

where 31 is 0x1f in hex

I understand from the commit comments that this was done for a good reason (an issue with live migration), but it's unfortunate that there isn't any control over it for a case like IGD passthrough.

Is it possible to somehow override this? Either by controlling how many PCI Bridges are created or by influencing the addresses they are created at? Perhaps a way to manipulate the command to KVM?

I tried looking for this file so I could patch it directly, but I wasn't able to. I presume it's compiled or somesuch? But if that's possible I'd be happy to go that route.

 

[sshaikh]

I edited the file at:

./usr/share/perl5/PVE/QemuServer.pm

and ran

pvedaemon restart

And have resolved the issue.

I commented out $bridges->{2} = 1; to avoid creating the second extra bridge. The alternative would be to keep it at a different address, but I feel that this might be safer, particularly as the bridge did not seem to be in use in this VM.

The only issue is that this will change the behaviour for all my VMs, which I'm a little uncomfortable with. Is there a way to only look for a VM id?

 

[sshaikh]

I've settled with the following patch:

    if (!$q35) {
        # add pci bridges
        if (min_version($machine_version, 2, 3)) {
           $bridges->{1} = 1;
           #Legacy IGD passthrough fix
           $bridges->{2} = 1 if $vmid != [VMID requiring passthrough];
        }

        $bridges->{3} = 1 if $scsihw =~ m/^virtio-scsi-single/;

    }

 

[n1nj4888]

With headless Intel NUCs (for Intel QuickSync purposes), I've only ever been able to get the Coffee Lake IGD passed through using i440fx - If I try q35, the Ubuntu VM always hangs during boot... I wonder if the above could explain that behaviour also?

Even so, if it's a bug, I think it would be better to raise it on https://bugzilla.proxmox.com so that it's understood / fixed for everyone's benefit?

Thanks!

 

[sshaikh]

The code specifically doesn't create the bridges with q35 (see above).

The VFIO guys insist that legacy passthrough will never work with q35, so it's probably something else. That said, they said the same about OVMF and it works there! But with coffee lake I think you can use UPT and not legacy anyway, which is the opposite (works only with q35).

EDIT: q35 seems to create it's own bridge at 1f.0: PCI: slot 31 function 0 not available for vfio-pci-igd-lpc-bridge, in use by ICH9-LPC. It appears that this can't be used by legacy passthrough, and isn't added by Proxmox.

Finally I don't think the addition of PCI bridges is a bug - it's pretty explicit and deliberate behaviour by Proxmox to support a feature. They themselves do not support IGD passthrough.

 

[sshaikh]

I went ahead and filed a bug: https://bugzilla.proxmox.com/show_bug.cgi?id=2794

 

[n1nj4888]

... But with coffee lake I think you can use UPT and not legacy anyway, which is the opposite (works only with q35).

Thanks for this. Is there a guide somewhere on the definition of and differences between UPT and legacy Passthrough and, importantly, how to configure each approach?

thanks!

 

[sshaikh]

This from the chap behind the IGD passthrough code in QEMU is very useful:

https://git.qemu.org/?p=qemu.git;a=...bb50789ada12b210897a5574bf89ee64b80fb;hb=HEAD