PCI passthrough count maximum?

[jbatter]

Hi,

I am trying to make use of a box with multiple GPU PCIE passthroughs to pass to a single VM. At this point, I have 7 GPUs, of which, I can get any 4 to correctly pass onto a VM, but I can't get more than that to pass. Is there a way to get more entries?

<< /etc/pve/nodes/host/qemu-server/101.conf >>:
Here's my config:
balloon: 2048
bootdisk: ide0
cores: 2
hostpci0: 08:00,pcie=1
#hostpci0: 84:00,pcie=1
hostpci1: 04:00,pcie=1
hostpci2: 05:00,pcie=1
hostpci3: 83:00,pcie=1
hostpci4: 84:00,pcie=1
hostpci5: 87:00,pcie=1
#hostpci6: 03:00,pcie=1
ide0: SSD-Raid0:vm-101-disk-1,size=32G
ide2: none,media=cdrom
machine: q35
memory: 16384
name: guest
net0: bridge=vmbr1,firewall=1,e1000=62:62:66:63:31:63
numa: 1
ostype: l26
smbios1: uuid=869e2c94-13c6-438d-b36f-30419d5e3556
sockets: 2


And here's the error I get when I try to pass the hostpci4 and hostpci5 entries (devices 5 and 6):
# qm start 101
vm 101 - unable to parse value of 'hostpci4' - unknown setting 'hostpci4'
vm 101 - unable to parse value of 'hostpci5' - unknown setting 'hostpci5'
vm 101 - unable to parse value of 'hostpci4' - unknown setting 'hostpci4'
vm 101 - unable to parse value of 'hostpci5' - unknown setting 'hostpci5'

 

[thenickdude]

I ran into this recently when trying to pass through an additional USB controller. Looks like passing through an extra 4 devices (total of 8) should be possible. I patched my Proxmox install to allow that and it's working for me, here's the instructions:

http://www.nicksherlock.com/2017/04/passthrough-more-than-4-pcie-devices-to-proxmox-4-4-guests/

 

[ONE FOTON]

Hello unfortunately did not work the patch in Proxmox 5.2.2

Will you know any other way to add more GPU to a VM?

 

[thenickdude]

I'm using the latest Proxmox 5.2 and the patch is working fine for me; I'm writing this reply from a VM with 6 passthrough PCIe devices (note, PCIe not PCI). You probably missed part of the patch.

 

[ONE FOTON]

Brother!
How did you do it?
You show me the conf of your VM?

 

[thenickdude]

I just made sure I had applied the patch correctly to those three files (/usr/share/qemu-server/pve-q35.cfg, /usr/share/perl5/PVE/QemuServer.pm, /usr/share/perl5/PVE/QemuServer/PCI.pm), and my VM configuration looks like:

hostpci0: 04:00,pcie=1,x-vga=on
hostpci1: 00:1a.0,pcie=1
hostpci2: 00:1d.0,pcie=1
hostpci3: 82:00.0,pcie=1
hostpci4: 81:00.0,pcie=1
hostpci5: 0b:00.0,pcie=1​

 

[ONE FOTON]

/usr/share/perl5/PVE/QemuServer.pm
Well this file is well modified, but do not match the lines.
It seems that my file is more full of code.
this line
my $MAX_HOSTPCI_DEVICES = 8;
It's in position 609, in my file.
Another problem I have had and is the Blessed Code 43 in the VM of win 10, I have been able to resolve with this Conf.
hostpci0: 04:00.0,x-vga=on,romfile=1rx480org.rom
hostpci1: 04:00.1
hostpci2: 05:00.0,x-vga=on,romfile=1-2rx480org.rom
hostpci3: 05:00.1
In other words they are GPUs AMD.

 

[thenickdude]

Don't bother checking for exact line numbers, just do a search for the text of the original line of code, and you should find it.

This config is incorrect:

hostpci0: 04:00.0,x-vga=on,romfile=1rx480org.rom
hostpci1: 04:00.1

Both lines should be replaced with just:

hostpci0: 04:00,x-vga=on,romfile=1rx480org.rom

That will pass both functions (the video card and the audio) as one device.

 

[ONE FOTON]

Hello brother!
After several days manage to make it work!
The problem was in the IOMMU groups!
The first card was inside the processor group!

I solved it by adding pcie_acs_override = downstream and updating the grub.
Also generate a new Initramfs.

So there I have mine them full!

 

[T.Herrmann]

Hello the configuration of the PCI.pm has changed in PVE 5.3. >>> /usr/share/perl5/PVE/QemuServer/PCI.pm

PCI.pm >> PVE 5.3

my $devices = {
vga => { bus => 'pcie.0', addr => 1 },
hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
# win7 is picky about pcie assignments
hostpci0bus0 => { bus => "pcie.0", addr => 16 },
hostpci1bus0 => { bus => "pcie.0", addr => 17 },
hostpci2bus0 => { bus => "pcie.0", addr => 18 },
hostpci3bus0 => { bus => "pcie.0", addr => 19 },

};

if (defined($devices->{$id}->{bus}) && defined($devices->{$id}->{addr})) {
my $addr = sprintf("0x%x", $devices->{$id}->{addr});


PCI.pm >> PVE 5.0, 5.1 , 5.2

my $devices = {
hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
};

if (defined($devices->{$id}->{bus}) && defined($devices->{$id}->{addr})) {
my $addr = sprintf("0x%x", $devices->{$id}->{addr});


See difference: thenickdude homepage and git repo

Are there any idea for modification to get 8 PCIE devices per VM?

 

[thenickdude]

I've updated that patch for 5.3 now:

https://github.com/thenickdude/qemu-server/commit/e1e221212cd7cc81d02277965d11f990b264eeaf