Unable to pass two VF of Intel X550-T2 as PCIe to two VMs

[Sandbo]

I have set up SR-IOV with my platform running PVE 6, with AMD Athlon 200GE, Asrock X470ITX and an Intel X550-T2. The guests here are two ubuntu server. Other combination of VMs were tested and produced the same result. Q35 and OVMF are being used.

While SR-IOV works, if I assign 1 VF each to 2 VMs with checking the box PCI-E, the 1st VM can boot but then the second VM will should "Guest has not initialized video" and stays there.

The same won't happen if I assign the VF as PCI without checking the PCIe box.

Is there anything I have missed?

 

[dcsapak]

can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?

 

[Sandbo]

can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?

Sure:

pveversion -v

root@pve:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-5.0.18-1-pve: 5.0.18-1
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-6
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

lspci

root@pve:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.6 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
01:10.0 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.2 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.4 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.6 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.7 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.0 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.1 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.2 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.3 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.4 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.5 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.6 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.7 Ethernet controller: Intel Corporation X550 Virtual Function
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
0b:00.0 Network controller: Intel Corporation Wireless-AC 9260 (rev 29)
0c:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
0d:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2262 (rev 03)
0e:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev cb)
0e:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller
0e:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
0e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
0e:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61)

iommu groupings.
note that the PFs are
01:00.0
01:00.1
the VFs are
01:10.0
01:10.1
01:10.2
01:10.3
01:10.4
01:10.5
01:10.6
01:10.7
01:11.0
01:11.1
01:11.2
01:11.3
01:11.4
01:11.5
01:11.6
01:11.7

root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:01:10.2
/sys/kernel/iommu_groups/7/devices/0000:00:14.3
/sys/kernel/iommu_groups/7/devices/0000:00:14.0
/sys/kernel/iommu_groups/25/devices/0000:01:10.3
/sys/kernel/iommu_groups/15/devices/0000:0f:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:08.1
/sys/kernel/iommu_groups/23/devices/0000:01:11.6
/sys/kernel/iommu_groups/13/devices/0000:0e:00.0
/sys/kernel/iommu_groups/31/devices/0000:01:11.7
/sys/kernel/iommu_groups/3/devices/0000:00:01.6
/sys/kernel/iommu_groups/21/devices/0000:01:11.2
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/11/devices/0000:0c:00.0
/sys/kernel/iommu_groups/11/devices/0000:04:06.0
/sys/kernel/iommu_groups/11/devices/0000:0b:00.0
/sys/kernel/iommu_groups/11/devices/0000:04:02.0
/sys/kernel/iommu_groups/11/devices/0000:04:05.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.1
/sys/kernel/iommu_groups/11/devices/0000:04:01.0
/sys/kernel/iommu_groups/11/devices/0000:04:04.0
/sys/kernel/iommu_groups/11/devices/0000:04:07.0
/sys/kernel/iommu_groups/11/devices/0000:04:00.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.2
/sys/kernel/iommu_groups/11/devices/0000:04:03.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:01:11.1
/sys/kernel/iommu_groups/18/devices/0000:01:10.4
/sys/kernel/iommu_groups/8/devices/0000:00:18.3
/sys/kernel/iommu_groups/8/devices/0000:00:18.1
/sys/kernel/iommu_groups/8/devices/0000:00:18.6
/sys/kernel/iommu_groups/8/devices/0000:00:18.4
/sys/kernel/iommu_groups/8/devices/0000:00:18.2
/sys/kernel/iommu_groups/8/devices/0000:00:18.0
/sys/kernel/iommu_groups/8/devices/0000:00:18.7
/sys/kernel/iommu_groups/8/devices/0000:00:18.5
/sys/kernel/iommu_groups/26/devices/0000:01:10.5
/sys/kernel/iommu_groups/16/devices/0000:01:10.0
/sys/kernel/iommu_groups/6/devices/0000:00:08.2
/sys/kernel/iommu_groups/24/devices/0000:01:10.1
/sys/kernel/iommu_groups/14/devices/0000:0e:00.3
/sys/kernel/iommu_groups/14/devices/0000:0e:00.1
/sys/kernel/iommu_groups/14/devices/0000:0e:00.4
/sys/kernel/iommu_groups/14/devices/0000:0e:00.2
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
/sys/kernel/iommu_groups/22/devices/0000:01:11.4
/sys/kernel/iommu_groups/12/devices/0000:0d:00.0
/sys/kernel/iommu_groups/30/devices/0000:01:11.5
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:01:11.0
/sys/kernel/iommu_groups/10/devices/0000:01:00.1
/sys/kernel/iommu_groups/29/devices/0000:01:11.3
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:01:10.6
/sys/kernel/iommu_groups/9/devices/0000:01:00.0
/sys/kernel/iommu_groups/27/devices/0000:01:10.7

finally, I have now three VMs:

100, router VM

agent: 1
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-100-disk-0,size=4M
hostpci0: 01:10.0
hostpci1: 01:10.1
ide2: none,media=cdrom
machine: q35
memory: 4096
name: CSQT
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=09b217a1-d055-4fd0-a2d4-9ff31b2a0bb7
sockets: 1
startup: order=1
vga: virtio
virtio0: local-lvm:vm-100-disk-1,size=32G
vmgenid: d601f53d-8fc8-42cb-88d8-baf41606f7ca

101, ubuntu 18.04 server 1

agent: 1
bios: ovmf
boot: cd
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 01:10.3
ide2: none,media=cdrom
machine: q35
memory: 4096
name: ubuntu-nextCloud
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=32b16ea9-b7b6-47a1-9155-cffaf0e53792
sockets: 1
startup: order=2,up=60
vga: virtio
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: 181d5cd3-6eb4-42b3-b401-b2cfbe5dd1af

102, ubuntu 18.04 server 2

agent: 0
bios: ovmf
boot: cd
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-102-disk-0,size=4M
hostpci0: 01:10.5
ide2: none,media=cdrom
machine: q35
memory: 4096
name: ubuntu-nextCloud
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=32b16ea9-b7b6-47a1-9155-cffaf0e53792
sockets: 1
startup: order=2,up=60
vga: virtio
virtio0: local-lvm:vm-102-disk-1,size=32G
vmgenid: cc1d98cf-d22c-43e0-a811-589df9309179

the assignment test has been done between any two VMs. As soon as I assign two PCI-E VFs over two VMs, the second one that boots later won't boot.

A similar experience is being shared by a redditor:
https://www.reddit.com/r/Proxmox/co...abling_sriov_for_intel_nic_x550t2_on/ew52gz6/

 

[Ming-Yuan Yu]

can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?

I am the redditor having similar issues.

# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-5.0.18-1-pve: 5.0.18-1
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-6
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

# lspci                 
00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 04)
00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
00:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 04)
00:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 04)
00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:11.0 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 0 (rev 08)
00:11.1 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 1 (rev 08)
00:11.5 SATA controller: Intel Corporation Lewisburg SSATA Controller [AHCI mode] (rev 08)
00:14.0 USB controller: Intel Corporation Lewisburg USB 3.0 xHCI Controller (rev 08)
00:14.2 Signal processing controller: Intel Corporation C620 Series Chipset Family Thermal Subsystem (rev 08)
00:16.0 Communication controller: Intel Corporation Lewisburg CSME: HECI #1 (rev 08)
00:16.1 Communication controller: Intel Corporation Lewisburg CSME: HECI #2 (rev 08)
00:16.4 Communication controller: Intel Corporation Lewisburg CSME: HECI #3 (rev 08)
00:17.0 SATA controller: Intel Corporation Lewisburg SATA Controller [AHCI mode] (rev 08)
00:1c.0 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #1 (rev f8)
00:1c.5 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #6 (rev f8)
00:1f.0 ISA bridge: Intel Corporation Lewisburg LPC Controller (rev 08)
00:1f.2 Memory controller: Intel Corporation Lewisburg PMC (rev 08)
00:1f.4 SMBus: Intel Corporation Lewisburg SMBus (rev 08)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller (rev 08)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
17:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04)
17:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
17:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
17:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
17:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
18:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
3a:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
3a:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
3a:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
3a:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
3a:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
3a:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
3a:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
3a:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
3a:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
3a:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
3a:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
3a:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
3a:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
3a:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
3a:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
3a:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
3a:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
5d:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04)
5d:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
5d:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
5d:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
5d:0e.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:0e.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:0f.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:0f.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:10.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:10.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5e:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 08)
5f:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 08)
60:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 08)
60:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 08)
60:02.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
80:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 04)
80:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 04)
80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
85:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
85:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
ae:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
ae:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
ae:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
ae:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
ae:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
ae:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
ae:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
ae:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
ae:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
ae:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
ae:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
ae:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
ae:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
ae:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
ae:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
ae:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
d7:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
d7:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
d7:0e.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:0e.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:0f.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:0f.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:10.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:10.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)

The PFs are
/sys/kernel/iommu_groups/44/devices/0000:60:00.0 >> NIC 1
/sys/kernel/iommu_groups/45/devices/0000:60:00.1 >> NIC 2

The VFs associated with NIC 1 are:
/sys/kernel/iommu_groups/75/devices/0000:60:02.0
/sys/kernel/iommu_groups/76/devices/0000:60:02.1
/sys/kernel/iommu_groups/77/devices/0000:60:02.2
/sys/kernel/iommu_groups/78/devices/0000:60:02.3

And the VFs associated with NIC 2 are:
/sys/kernel/iommu_groups/79/devices/0000:60:06.0
/sys/kernel/iommu_groups/80/devices/0000:60:06.1
/sys/kernel/iommu_groups/81/devices/0000:60:06.2
/sys/kernel/iommu_groups/82/devices/0000:60:06.3

The config for the VM is

bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: local-zfs:vm-901-disk-1,size=128K
hostpci0: 60:06.0,pcie=1
hostpci1: 60:06.1
hostpci2: 60:06.2
hostpci3: 60:06.3
machine: q35
memory: 16384
name: test01
numa: 1
ostype: l26
sata0: local:iso/ubuntu-18.04.2-desktop-amd64.iso,media=cdrom,size=1904M
scsi0: local-zfs:vm-901-disk-0,cache=writeback,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=024b8ccd-bc78-4cbb-aaf4-a32c8652636c
sockets: 2
vga: virtio
vmgenid: 19fcf93d-01be-4c1e-8afc-7344ff95fc85

My problem is a little bit different thought. Only the first VF on each NIC can be set to PCIe mode. Unlike the original post, I can passthrough multiple VFs to a VM, but only the first one can be in PCIe mode. The other VFs simply does not work with `pcie=1` even if they are the only VF in the VM.

 

[Ming-Yuan Yu]

Here is the full list of IOMMU groups (which is too long to fit into one reply)

# find /sys/kernel/iommu_groups/ -type l | sort
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/12/devices/0000:03:00.0
/sys/kernel/iommu_groups/13/devices/0000:17:00.0
/sys/kernel/iommu_groups/14/devices/0000:17:05.0
/sys/kernel/iommu_groups/14/devices/0000:17:05.2
/sys/kernel/iommu_groups/14/devices/0000:17:05.4
/sys/kernel/iommu_groups/15/devices/0000:17:08.0
/sys/kernel/iommu_groups/15/devices/0000:17:08.1
/sys/kernel/iommu_groups/15/devices/0000:17:08.2
/sys/kernel/iommu_groups/15/devices/0000:17:08.3
/sys/kernel/iommu_groups/15/devices/0000:17:08.4
/sys/kernel/iommu_groups/15/devices/0000:17:08.5
/sys/kernel/iommu_groups/15/devices/0000:17:08.6
/sys/kernel/iommu_groups/15/devices/0000:17:08.7
/sys/kernel/iommu_groups/16/devices/0000:17:09.0
/sys/kernel/iommu_groups/16/devices/0000:17:09.1
/sys/kernel/iommu_groups/16/devices/0000:17:09.2
/sys/kernel/iommu_groups/16/devices/0000:17:09.3
/sys/kernel/iommu_groups/16/devices/0000:17:09.4
/sys/kernel/iommu_groups/16/devices/0000:17:09.5
/sys/kernel/iommu_groups/16/devices/0000:17:09.6
/sys/kernel/iommu_groups/16/devices/0000:17:09.7
/sys/kernel/iommu_groups/17/devices/0000:17:0a.0
/sys/kernel/iommu_groups/17/devices/0000:17:0a.1
/sys/kernel/iommu_groups/17/devices/0000:17:0a.2
/sys/kernel/iommu_groups/17/devices/0000:17:0a.3
/sys/kernel/iommu_groups/17/devices/0000:17:0a.4
/sys/kernel/iommu_groups/17/devices/0000:17:0a.5
/sys/kernel/iommu_groups/17/devices/0000:17:0a.6
/sys/kernel/iommu_groups/17/devices/0000:17:0a.7
/sys/kernel/iommu_groups/18/devices/0000:17:0b.0
/sys/kernel/iommu_groups/18/devices/0000:17:0b.1
/sys/kernel/iommu_groups/18/devices/0000:17:0b.2
/sys/kernel/iommu_groups/18/devices/0000:17:0b.3
/sys/kernel/iommu_groups/19/devices/0000:17:0e.0
/sys/kernel/iommu_groups/19/devices/0000:17:0e.1
/sys/kernel/iommu_groups/19/devices/0000:17:0e.2
/sys/kernel/iommu_groups/19/devices/0000:17:0e.3
/sys/kernel/iommu_groups/19/devices/0000:17:0e.4
/sys/kernel/iommu_groups/19/devices/0000:17:0e.5
/sys/kernel/iommu_groups/19/devices/0000:17:0e.6
/sys/kernel/iommu_groups/19/devices/0000:17:0e.7
/sys/kernel/iommu_groups/1/devices/0000:00:04.0
/sys/kernel/iommu_groups/1/devices/0000:00:04.1
/sys/kernel/iommu_groups/1/devices/0000:00:04.2
/sys/kernel/iommu_groups/1/devices/0000:00:04.3
/sys/kernel/iommu_groups/1/devices/0000:00:04.4
/sys/kernel/iommu_groups/1/devices/0000:00:04.5
/sys/kernel/iommu_groups/1/devices/0000:00:04.6
/sys/kernel/iommu_groups/1/devices/0000:00:04.7
/sys/kernel/iommu_groups/20/devices/0000:17:0f.0
/sys/kernel/iommu_groups/20/devices/0000:17:0f.1
/sys/kernel/iommu_groups/20/devices/0000:17:0f.2
/sys/kernel/iommu_groups/20/devices/0000:17:0f.3
/sys/kernel/iommu_groups/20/devices/0000:17:0f.4
/sys/kernel/iommu_groups/20/devices/0000:17:0f.5
/sys/kernel/iommu_groups/20/devices/0000:17:0f.6
/sys/kernel/iommu_groups/20/devices/0000:17:0f.7
/sys/kernel/iommu_groups/21/devices/0000:17:10.0
/sys/kernel/iommu_groups/21/devices/0000:17:10.1
/sys/kernel/iommu_groups/21/devices/0000:17:10.2
/sys/kernel/iommu_groups/21/devices/0000:17:10.3
/sys/kernel/iommu_groups/21/devices/0000:17:10.4
/sys/kernel/iommu_groups/21/devices/0000:17:10.5
/sys/kernel/iommu_groups/21/devices/0000:17:10.6
/sys/kernel/iommu_groups/21/devices/0000:17:10.7
/sys/kernel/iommu_groups/22/devices/0000:17:11.0
/sys/kernel/iommu_groups/22/devices/0000:17:11.1
/sys/kernel/iommu_groups/22/devices/0000:17:11.2
/sys/kernel/iommu_groups/22/devices/0000:17:11.3
/sys/kernel/iommu_groups/23/devices/0000:17:1d.0
/sys/kernel/iommu_groups/23/devices/0000:17:1d.1
/sys/kernel/iommu_groups/23/devices/0000:17:1d.2
/sys/kernel/iommu_groups/23/devices/0000:17:1d.3
/sys/kernel/iommu_groups/24/devices/0000:17:1e.0
/sys/kernel/iommu_groups/24/devices/0000:17:1e.1
/sys/kernel/iommu_groups/24/devices/0000:17:1e.2
/sys/kernel/iommu_groups/24/devices/0000:17:1e.3
/sys/kernel/iommu_groups/24/devices/0000:17:1e.4
/sys/kernel/iommu_groups/24/devices/0000:17:1e.5
/sys/kernel/iommu_groups/24/devices/0000:17:1e.6
/sys/kernel/iommu_groups/25/devices/0000:18:00.0
/sys/kernel/iommu_groups/26/devices/0000:3a:05.0
/sys/kernel/iommu_groups/26/devices/0000:3a:05.2
/sys/kernel/iommu_groups/26/devices/0000:3a:05.4
/sys/kernel/iommu_groups/27/devices/0000:3a:08.0
/sys/kernel/iommu_groups/28/devices/0000:3a:09.0
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.0
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.1
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.2
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.3
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.4
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.5
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.6
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.7
/sys/kernel/iommu_groups/2/devices/0000:00:05.0
/sys/kernel/iommu_groups/2/devices/0000:00:05.2
/sys/kernel/iommu_groups/2/devices/0000:00:05.4
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.0
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.1
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.2
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.3
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.0
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.1
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.2
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.3
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.4
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.5
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.6
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.7
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.0
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.1
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.2
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.3
/sys/kernel/iommu_groups/33/devices/0000:5d:02.0
/sys/kernel/iommu_groups/34/devices/0000:5d:05.0
/sys/kernel/iommu_groups/34/devices/0000:5d:05.2
/sys/kernel/iommu_groups/34/devices/0000:5d:05.4
/sys/kernel/iommu_groups/35/devices/0000:5d:0e.0
/sys/kernel/iommu_groups/35/devices/0000:5d:0e.1
/sys/kernel/iommu_groups/36/devices/0000:5d:0f.0
/sys/kernel/iommu_groups/36/devices/0000:5d:0f.1
/sys/kernel/iommu_groups/37/devices/0000:5d:10.0
/sys/kernel/iommu_groups/37/devices/0000:5d:10.1
/sys/kernel/iommu_groups/38/devices/0000:5d:12.0
/sys/kernel/iommu_groups/38/devices/0000:5d:12.1
/sys/kernel/iommu_groups/38/devices/0000:5d:12.2
/sys/kernel/iommu_groups/38/devices/0000:5d:12.4
/sys/kernel/iommu_groups/38/devices/0000:5d:12.5
/sys/kernel/iommu_groups/39/devices/0000:5d:15.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.1
/sys/kernel/iommu_groups/3/devices/0000:00:08.2
/sys/kernel/iommu_groups/40/devices/0000:5d:16.0
/sys/kernel/iommu_groups/40/devices/0000:5d:16.4
/sys/kernel/iommu_groups/41/devices/0000:5d:17.0
/sys/kernel/iommu_groups/42/devices/0000:5e:00.0
/sys/kernel/iommu_groups/43/devices/0000:5f:03.0
/sys/kernel/iommu_groups/44/devices/0000:60:00.0
/sys/kernel/iommu_groups/45/devices/0000:60:00.1
/sys/kernel/iommu_groups/46/devices/0000:80:04.0
/sys/kernel/iommu_groups/46/devices/0000:80:04.1
/sys/kernel/iommu_groups/46/devices/0000:80:04.2
/sys/kernel/iommu_groups/46/devices/0000:80:04.3
/sys/kernel/iommu_groups/46/devices/0000:80:04.4
/sys/kernel/iommu_groups/46/devices/0000:80:04.5
/sys/kernel/iommu_groups/46/devices/0000:80:04.6
/sys/kernel/iommu_groups/46/devices/0000:80:04.7
/sys/kernel/iommu_groups/47/devices/0000:80:05.0
/sys/kernel/iommu_groups/47/devices/0000:80:05.2
/sys/kernel/iommu_groups/47/devices/0000:80:05.4
/sys/kernel/iommu_groups/48/devices/0000:80:08.0
/sys/kernel/iommu_groups/48/devices/0000:80:08.1
/sys/kernel/iommu_groups/48/devices/0000:80:08.2
/sys/kernel/iommu_groups/49/devices/0000:85:05.0
/sys/kernel/iommu_groups/49/devices/0000:85:05.2
/sys/kernel/iommu_groups/49/devices/0000:85:05.4
/sys/kernel/iommu_groups/4/devices/0000:00:11.0
/sys/kernel/iommu_groups/4/devices/0000:00:11.1
/sys/kernel/iommu_groups/4/devices/0000:00:11.5
/sys/kernel/iommu_groups/50/devices/0000:85:08.0
/sys/kernel/iommu_groups/50/devices/0000:85:08.1
/sys/kernel/iommu_groups/50/devices/0000:85:08.2
/sys/kernel/iommu_groups/50/devices/0000:85:08.3
/sys/kernel/iommu_groups/50/devices/0000:85:08.4
/sys/kernel/iommu_groups/50/devices/0000:85:08.5
/sys/kernel/iommu_groups/50/devices/0000:85:08.6
/sys/kernel/iommu_groups/50/devices/0000:85:08.7
/sys/kernel/iommu_groups/51/devices/0000:85:09.0
/sys/kernel/iommu_groups/51/devices/0000:85:09.1
/sys/kernel/iommu_groups/51/devices/0000:85:09.2
/sys/kernel/iommu_groups/51/devices/0000:85:09.3
/sys/kernel/iommu_groups/51/devices/0000:85:09.4
/sys/kernel/iommu_groups/51/devices/0000:85:09.5
/sys/kernel/iommu_groups/51/devices/0000:85:09.6
/sys/kernel/iommu_groups/51/devices/0000:85:09.7
/sys/kernel/iommu_groups/52/devices/0000:85:0a.0
/sys/kernel/iommu_groups/52/devices/0000:85:0a.1
/sys/kernel/iommu_groups/52/devices/0000:85:0a.2
/sys/kernel/iommu_groups/52/devices/0000:85:0a.3
/sys/kernel/iommu_groups/52/devices/0000:85:0a.4
/sys/kernel/iommu_groups/52/devices/0000:85:0a.5
/sys/kernel/iommu_groups/52/devices/0000:85:0a.6
/sys/kernel/iommu_groups/52/devices/0000:85:0a.7
/sys/kernel/iommu_groups/53/devices/0000:85:0b.0
/sys/kernel/iommu_groups/53/devices/0000:85:0b.1
/sys/kernel/iommu_groups/53/devices/0000:85:0b.2
/sys/kernel/iommu_groups/53/devices/0000:85:0b.3
/sys/kernel/iommu_groups/54/devices/0000:85:0e.0
/sys/kernel/iommu_groups/54/devices/0000:85:0e.1
/sys/kernel/iommu_groups/54/devices/0000:85:0e.2
/sys/kernel/iommu_groups/54/devices/0000:85:0e.3
/sys/kernel/iommu_groups/54/devices/0000:85:0e.4
/sys/kernel/iommu_groups/54/devices/0000:85:0e.5
/sys/kernel/iommu_groups/54/devices/0000:85:0e.6
/sys/kernel/iommu_groups/54/devices/0000:85:0e.7
/sys/kernel/iommu_groups/55/devices/0000:85:0f.0
/sys/kernel/iommu_groups/55/devices/0000:85:0f.1
/sys/kernel/iommu_groups/55/devices/0000:85:0f.2
/sys/kernel/iommu_groups/55/devices/0000:85:0f.3
/sys/kernel/iommu_groups/55/devices/0000:85:0f.4
/sys/kernel/iommu_groups/55/devices/0000:85:0f.5
/sys/kernel/iommu_groups/55/devices/0000:85:0f.6
/sys/kernel/iommu_groups/55/devices/0000:85:0f.7
/sys/kernel/iommu_groups/56/devices/0000:85:10.0
/sys/kernel/iommu_groups/56/devices/0000:85:10.1
/sys/kernel/iommu_groups/56/devices/0000:85:10.2
/sys/kernel/iommu_groups/56/devices/0000:85:10.3
/sys/kernel/iommu_groups/56/devices/0000:85:10.4
/sys/kernel/iommu_groups/56/devices/0000:85:10.5
/sys/kernel/iommu_groups/56/devices/0000:85:10.6
/sys/kernel/iommu_groups/56/devices/0000:85:10.7
/sys/kernel/iommu_groups/57/devices/0000:85:11.0
/sys/kernel/iommu_groups/57/devices/0000:85:11.1
/sys/kernel/iommu_groups/57/devices/0000:85:11.2
/sys/kernel/iommu_groups/57/devices/0000:85:11.3
/sys/kernel/iommu_groups/58/devices/0000:85:1d.0
/sys/kernel/iommu_groups/58/devices/0000:85:1d.1
/sys/kernel/iommu_groups/58/devices/0000:85:1d.2
/sys/kernel/iommu_groups/58/devices/0000:85:1d.3
/sys/kernel/iommu_groups/59/devices/0000:85:1e.0
/sys/kernel/iommu_groups/59/devices/0000:85:1e.1
/sys/kernel/iommu_groups/59/devices/0000:85:1e.2
/sys/kernel/iommu_groups/59/devices/0000:85:1e.3
/sys/kernel/iommu_groups/59/devices/0000:85:1e.4
/sys/kernel/iommu_groups/59/devices/0000:85:1e.5
/sys/kernel/iommu_groups/59/devices/0000:85:1e.6
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/60/devices/0000:ae:05.0
/sys/kernel/iommu_groups/60/devices/0000:ae:05.2
/sys/kernel/iommu_groups/60/devices/0000:ae:05.4
/sys/kernel/iommu_groups/61/devices/0000:ae:08.0
/sys/kernel/iommu_groups/62/devices/0000:ae:09.0
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.0
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.1
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.2
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.3
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.4
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.5
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.6
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.7
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.0
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.1
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.2
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.3
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.0
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.1
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.2
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.3
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.4
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.5
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.6
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.7
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.0
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.1
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.2
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.3
/sys/kernel/iommu_groups/67/devices/0000:d7:05.0
/sys/kernel/iommu_groups/67/devices/0000:d7:05.2
/sys/kernel/iommu_groups/67/devices/0000:d7:05.4
/sys/kernel/iommu_groups/68/devices/0000:d7:0e.0
/sys/kernel/iommu_groups/68/devices/0000:d7:0e.1
/sys/kernel/iommu_groups/69/devices/0000:d7:0f.0
/sys/kernel/iommu_groups/69/devices/0000:d7:0f.1
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:16.1
/sys/kernel/iommu_groups/6/devices/0000:00:16.4
/sys/kernel/iommu_groups/70/devices/0000:d7:10.0
/sys/kernel/iommu_groups/70/devices/0000:d7:10.1
/sys/kernel/iommu_groups/71/devices/0000:d7:12.0
/sys/kernel/iommu_groups/71/devices/0000:d7:12.1
/sys/kernel/iommu_groups/71/devices/0000:d7:12.2
/sys/kernel/iommu_groups/71/devices/0000:d7:12.4
/sys/kernel/iommu_groups/71/devices/0000:d7:12.5
/sys/kernel/iommu_groups/72/devices/0000:d7:15.0
/sys/kernel/iommu_groups/73/devices/0000:d7:16.0
/sys/kernel/iommu_groups/73/devices/0000:d7:16.4
/sys/kernel/iommu_groups/74/devices/0000:d7:17.0
/sys/kernel/iommu_groups/75/devices/0000:60:02.0
/sys/kernel/iommu_groups/76/devices/0000:60:02.1
/sys/kernel/iommu_groups/77/devices/0000:60:02.2
/sys/kernel/iommu_groups/78/devices/0000:60:02.3
/sys/kernel/iommu_groups/79/devices/0000:60:06.0
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
/sys/kernel/iommu_groups/80/devices/0000:60:06.1
/sys/kernel/iommu_groups/81/devices/0000:60:06.2
/sys/kernel/iommu_groups/82/devices/0000:60:06.3
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.5

 

[dcsapak]

could you both try to set the machine type to 'pc-q35-3.1'

there were some changes in qemu 4 (which should be fixed with 4.0.1 or 4.1, depending what we will package) that can be
fixed with temporarily changing to a 3.1 machine type

 

[Ming-Yuan Yu]

could you both try to set the machine type to 'pc-q35-3.1'

there were some changes in qemu 4 (which should be fixed with 4.0.1 or 4.1, depending what we will package) that can be
fixed with temporarily changing to a 3.1 machine type

Setting machine type to `pc-q35-3.1` does not seem to change the behavior. It still freezes at `Guest has not initialized the display (yet)` when setting `pcie=1` on other VFs.

I forget to mention that my NICs are Intel X722.

 

[cryse]

I have the same issue, cannot start second VM if pcie=1 enabled.

 

[dcsapak]

could you try the newer kernel from the repos (5.3)?
maybe it is a driver issue...

i have here an I350 with sr-iov, and i can passthrough multiple vfs in either pci/pcie mode without problems....