Can only pass-through one GPU device ID. 4 total in system

[p.lakis]

So i have 4x 1070s in the system, device-ids - 3:00, 4:00, 5:00 & 83:00.

I can pass-through 83:00 without any issues. This is the device id i chose to start the configuration with and since then can pass through another device-id.
If i remove 83:00 and add any other device-id it will not work.

I have a Win10 and Ubuntu16 + 18 VMs working with 83:00 device-id

All vendor ids are identical. My /etc/modprobe.d/vfio.conf

 options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1

All GPUs have

 Kernel driver in use: vfio-pci

Am i missing something in the vfio.conf? Do i need to add all device-ids + vendor-ids?

Thanks

 

[dcsapak]

how do your iommu groups look like ?

d add any other device-id it will not work.

what exactly happens? error when starting (please post the error)? no display output (please post dmesg output)?

 

[p.lakis]

what exactly happens? error when starting (please post the error)? no display output (please post dmesg output)?

No errors.
The pcie device doesnt show in hardware unless its device-id 83:00
If i pass multiple through including 83:00 only 83:00 shows in the OS, if i remove 83:00 and add any of the others they dont show up at all and i have access through noVNC through the web console. Which doesnt happen if 83:00 is there.

 

[dcsapak]

can you post the vm config?

 

[p.lakis]

can you post the vm config?

agent: 1
bios: ovmf
bootdisk: scsi0
cores: 4
efidisk0: SNVM-Data:vm-113-disk-2,size=128K
ide0: iso:iso/virtio-win-0.1.149.iso,media=cdrom,size=316634K
ide2: iso:iso/Windows.iso,media=cdrom
machine: q35
memory: 7000
name: multi
net0: virtio=2A:86:86:38:AF:54,bridge=vmbr1
numa: 1
ostype: win10
scsi0: SNVM-Data:vm-113-disk-1,cache=writeback,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=61893b19-b871-48d6-bf4a-85efa6b7c7e7
sockets: 2
hostpci0: 3:00,pcie=1,x-vga=on

If the last line is hostpci0: 83:00,pcie=1,x-vga=on everything works and PCI device shows in hardware, but with the above vm config the pci device does not show in hardware or OS. No errors.

 

[dcsapak]

hostpci0: 3:00,pcie=1,x-vga=on

has to be

hostpci0: 03:00,pcie=1,x-vga=on

the leading 0 is necessary