Unable to pass two VF of Intel X550-T2 as PCIe to two VMs

Sandbo

New Member
Jul 4, 2019
22
1
3
29
I have set up SR-IOV with my platform running PVE 6, with AMD Athlon 200GE, Asrock X470ITX and an Intel X550-T2. The guests here are two ubuntu server. Other combination of VMs were tested and produced the same result. Q35 and OVMF are being used.

While SR-IOV works, if I assign 1 VF each to 2 VMs with checking the box PCI-E, the 1st VM can boot but then the second VM will should "Guest has not initialized video" and stays there.

The same won't happen if I assign the VF as PCI without checking the PCIe box.

Is there anything I have missed?
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
4,074
368
88
31
Vienna
can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?
 

Sandbo

New Member
Jul 4, 2019
22
1
3
29
can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?
Sure:

pveversion -v

Code:
root@pve:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-5.0.18-1-pve: 5.0.18-1
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-6
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1
lspci

Code:
root@pve:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.6 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
01:10.0 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.2 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.4 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.6 Ethernet controller: Intel Corporation X550 Virtual Function
01:10.7 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.0 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.1 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.2 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.3 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.4 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.5 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.6 Ethernet controller: Intel Corporation X550 Virtual Function
01:11.7 Ethernet controller: Intel Corporation X550 Virtual Function
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43d0 (rev 01)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
04:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
0b:00.0 Network controller: Intel Corporation Wireless-AC 9260 (rev 29)
0c:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
0d:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2262 (rev 03)
0e:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series] (rev cb)
0e:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller
0e:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
0e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
0e:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61)
iommu groupings.
note that the PFs are
01:00.0
01:00.1
the VFs are
01:10.0
01:10.1
01:10.2
01:10.3
01:10.4
01:10.5
01:10.6
01:10.7
01:11.0
01:11.1
01:11.2
01:11.3
01:11.4
01:11.5
01:11.6
01:11.7
Code:
root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:01:10.2
/sys/kernel/iommu_groups/7/devices/0000:00:14.3
/sys/kernel/iommu_groups/7/devices/0000:00:14.0
/sys/kernel/iommu_groups/25/devices/0000:01:10.3
/sys/kernel/iommu_groups/15/devices/0000:0f:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:08.1
/sys/kernel/iommu_groups/23/devices/0000:01:11.6
/sys/kernel/iommu_groups/13/devices/0000:0e:00.0
/sys/kernel/iommu_groups/31/devices/0000:01:11.7
/sys/kernel/iommu_groups/3/devices/0000:00:01.6
/sys/kernel/iommu_groups/21/devices/0000:01:11.2
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/11/devices/0000:0c:00.0
/sys/kernel/iommu_groups/11/devices/0000:04:06.0
/sys/kernel/iommu_groups/11/devices/0000:0b:00.0
/sys/kernel/iommu_groups/11/devices/0000:04:02.0
/sys/kernel/iommu_groups/11/devices/0000:04:05.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.1
/sys/kernel/iommu_groups/11/devices/0000:04:01.0
/sys/kernel/iommu_groups/11/devices/0000:04:04.0
/sys/kernel/iommu_groups/11/devices/0000:04:07.0
/sys/kernel/iommu_groups/11/devices/0000:04:00.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.2
/sys/kernel/iommu_groups/11/devices/0000:04:03.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:01:11.1
/sys/kernel/iommu_groups/18/devices/0000:01:10.4
/sys/kernel/iommu_groups/8/devices/0000:00:18.3
/sys/kernel/iommu_groups/8/devices/0000:00:18.1
/sys/kernel/iommu_groups/8/devices/0000:00:18.6
/sys/kernel/iommu_groups/8/devices/0000:00:18.4
/sys/kernel/iommu_groups/8/devices/0000:00:18.2
/sys/kernel/iommu_groups/8/devices/0000:00:18.0
/sys/kernel/iommu_groups/8/devices/0000:00:18.7
/sys/kernel/iommu_groups/8/devices/0000:00:18.5
/sys/kernel/iommu_groups/26/devices/0000:01:10.5
/sys/kernel/iommu_groups/16/devices/0000:01:10.0
/sys/kernel/iommu_groups/6/devices/0000:00:08.2
/sys/kernel/iommu_groups/24/devices/0000:01:10.1
/sys/kernel/iommu_groups/14/devices/0000:0e:00.3
/sys/kernel/iommu_groups/14/devices/0000:0e:00.1
/sys/kernel/iommu_groups/14/devices/0000:0e:00.4
/sys/kernel/iommu_groups/14/devices/0000:0e:00.2
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
/sys/kernel/iommu_groups/22/devices/0000:01:11.4
/sys/kernel/iommu_groups/12/devices/0000:0d:00.0
/sys/kernel/iommu_groups/30/devices/0000:01:11.5
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:01:11.0
/sys/kernel/iommu_groups/10/devices/0000:01:00.1
/sys/kernel/iommu_groups/29/devices/0000:01:11.3
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:01:10.6
/sys/kernel/iommu_groups/9/devices/0000:01:00.0
/sys/kernel/iommu_groups/27/devices/0000:01:10.7
finally, I have now three VMs:

100, router VM

Code:
agent: 1
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-100-disk-0,size=4M
hostpci0: 01:10.0
hostpci1: 01:10.1
ide2: none,media=cdrom
machine: q35
memory: 4096
name: CSQT
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=09b217a1-d055-4fd0-a2d4-9ff31b2a0bb7
sockets: 1
startup: order=1
vga: virtio
virtio0: local-lvm:vm-100-disk-1,size=32G
vmgenid: d601f53d-8fc8-42cb-88d8-baf41606f7ca
101, ubuntu 18.04 server 1
Code:
agent: 1
bios: ovmf
boot: cd
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-101-disk-0,size=4M
hostpci0: 01:10.3
ide2: none,media=cdrom
machine: q35
memory: 4096
name: ubuntu-nextCloud
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=32b16ea9-b7b6-47a1-9155-cffaf0e53792
sockets: 1
startup: order=2,up=60
vga: virtio
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: 181d5cd3-6eb4-42b3-b401-b2cfbe5dd1af
102, ubuntu 18.04 server 2
Code:
agent: 0
bios: ovmf
boot: cd
bootdisk: virtio0
cores: 4
cpu: host
efidisk0: local-lvm:vm-102-disk-0,size=4M
hostpci0: 01:10.5
ide2: none,media=cdrom
machine: q35
memory: 4096
name: ubuntu-nextCloud
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=32b16ea9-b7b6-47a1-9155-cffaf0e53792
sockets: 1
startup: order=2,up=60
vga: virtio
virtio0: local-lvm:vm-102-disk-1,size=32G
vmgenid: cc1d98cf-d22c-43e0-a811-589df9309179
the assignment test has been done between any two VMs. As soon as I assign two PCI-E VFs over two VMs, the second one that boots later won't boot.

A similar experience is being shared by a redditor:
https://www.reddit.com/r/Proxmox/comments/cm81tc/tutorial_enabling_sriov_for_intel_nic_x550t2_on/ew52gz6/
 

Ming-Yuan Yu

New Member
Aug 8, 2019
3
0
1
28
can you post your 'pveversion -v' and 'lspci' output, the vm configs and the iommu groupings ?
I am the redditor having similar issues.

Code:
# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-5.0.18-1-pve: 5.0.18-1
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-6
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1
Code:
# lspci                 
00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 04)
00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
00:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 04)
00:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 04)
00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
00:11.0 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 0 (rev 08)
00:11.1 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 1 (rev 08)
00:11.5 SATA controller: Intel Corporation Lewisburg SSATA Controller [AHCI mode] (rev 08)
00:14.0 USB controller: Intel Corporation Lewisburg USB 3.0 xHCI Controller (rev 08)
00:14.2 Signal processing controller: Intel Corporation C620 Series Chipset Family Thermal Subsystem (rev 08)
00:16.0 Communication controller: Intel Corporation Lewisburg CSME: HECI #1 (rev 08)
00:16.1 Communication controller: Intel Corporation Lewisburg CSME: HECI #2 (rev 08)
00:16.4 Communication controller: Intel Corporation Lewisburg CSME: HECI #3 (rev 08)
00:17.0 SATA controller: Intel Corporation Lewisburg SATA Controller [AHCI mode] (rev 08)
00:1c.0 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #1 (rev f8)
00:1c.5 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #6 (rev f8)
00:1f.0 ISA bridge: Intel Corporation Lewisburg LPC Controller (rev 08)
00:1f.2 Memory controller: Intel Corporation Lewisburg PMC (rev 08)
00:1f.4 SMBus: Intel Corporation Lewisburg SMBus (rev 08)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller (rev 08)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
17:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04)
17:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
17:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
17:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
17:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
17:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
17:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
18:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
3a:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
3a:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
3a:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
3a:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
3a:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
3a:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
3a:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
3a:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
3a:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
3a:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
3a:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
3a:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
3a:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
3a:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
3a:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
3a:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
3a:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
3a:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
5d:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04)
5d:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
5d:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
5d:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
5d:0e.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:0e.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:0f.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:0f.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:10.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
5d:10.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
5d:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
5d:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5d:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
5e:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 08)
5f:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 08)
60:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 08)
60:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 08)
60:02.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:02.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
60:06.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 08)
80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
80:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 04)
80:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 04)
80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
85:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
85:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
ae:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
ae:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
ae:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
ae:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
ae:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
ae:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
ae:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
ae:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
ae:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
ae:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 04)
ae:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 04)
ae:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 04)
ae:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 04)
ae:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 04)
ae:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 04)
ae:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 04)
ae:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 04)
d7:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 04)
d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
d7:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 04)
d7:0e.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:0e.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:0f.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:0f.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:10.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 04)
d7:10.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 04)
d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
d7:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
The PFs are
/sys/kernel/iommu_groups/44/devices/0000:60:00.0 >> NIC 1
/sys/kernel/iommu_groups/45/devices/0000:60:00.1 >> NIC 2

The VFs associated with NIC 1 are:
/sys/kernel/iommu_groups/75/devices/0000:60:02.0
/sys/kernel/iommu_groups/76/devices/0000:60:02.1
/sys/kernel/iommu_groups/77/devices/0000:60:02.2
/sys/kernel/iommu_groups/78/devices/0000:60:02.3

And the VFs associated with NIC 2 are:
/sys/kernel/iommu_groups/79/devices/0000:60:06.0
/sys/kernel/iommu_groups/80/devices/0000:60:06.1
/sys/kernel/iommu_groups/81/devices/0000:60:06.2
/sys/kernel/iommu_groups/82/devices/0000:60:06.3

The config for the VM is
Code:
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: local-zfs:vm-901-disk-1,size=128K
hostpci0: 60:06.0,pcie=1
hostpci1: 60:06.1
hostpci2: 60:06.2
hostpci3: 60:06.3
machine: q35
memory: 16384
name: test01
numa: 1
ostype: l26
sata0: local:iso/ubuntu-18.04.2-desktop-amd64.iso,media=cdrom,size=1904M
scsi0: local-zfs:vm-901-disk-0,cache=writeback,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=024b8ccd-bc78-4cbb-aaf4-a32c8652636c
sockets: 2
vga: virtio
vmgenid: 19fcf93d-01be-4c1e-8afc-7344ff95fc85
My problem is a little bit different thought. Only the first VF on each NIC can be set to PCIe mode. Unlike the original post, I can passthrough multiple VFs to a VM, but only the first one can be in PCIe mode. The other VFs simply does not work with `pcie=1` even if they are the only VF in the VM.
 

Ming-Yuan Yu

New Member
Aug 8, 2019
3
0
1
28
Here is the full list of IOMMU groups (which is too long to fit into one reply)
Code:
# find /sys/kernel/iommu_groups/ -type l | sort
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/12/devices/0000:03:00.0
/sys/kernel/iommu_groups/13/devices/0000:17:00.0
/sys/kernel/iommu_groups/14/devices/0000:17:05.0
/sys/kernel/iommu_groups/14/devices/0000:17:05.2
/sys/kernel/iommu_groups/14/devices/0000:17:05.4
/sys/kernel/iommu_groups/15/devices/0000:17:08.0
/sys/kernel/iommu_groups/15/devices/0000:17:08.1
/sys/kernel/iommu_groups/15/devices/0000:17:08.2
/sys/kernel/iommu_groups/15/devices/0000:17:08.3
/sys/kernel/iommu_groups/15/devices/0000:17:08.4
/sys/kernel/iommu_groups/15/devices/0000:17:08.5
/sys/kernel/iommu_groups/15/devices/0000:17:08.6
/sys/kernel/iommu_groups/15/devices/0000:17:08.7
/sys/kernel/iommu_groups/16/devices/0000:17:09.0
/sys/kernel/iommu_groups/16/devices/0000:17:09.1
/sys/kernel/iommu_groups/16/devices/0000:17:09.2
/sys/kernel/iommu_groups/16/devices/0000:17:09.3
/sys/kernel/iommu_groups/16/devices/0000:17:09.4
/sys/kernel/iommu_groups/16/devices/0000:17:09.5
/sys/kernel/iommu_groups/16/devices/0000:17:09.6
/sys/kernel/iommu_groups/16/devices/0000:17:09.7
/sys/kernel/iommu_groups/17/devices/0000:17:0a.0
/sys/kernel/iommu_groups/17/devices/0000:17:0a.1
/sys/kernel/iommu_groups/17/devices/0000:17:0a.2
/sys/kernel/iommu_groups/17/devices/0000:17:0a.3
/sys/kernel/iommu_groups/17/devices/0000:17:0a.4
/sys/kernel/iommu_groups/17/devices/0000:17:0a.5
/sys/kernel/iommu_groups/17/devices/0000:17:0a.6
/sys/kernel/iommu_groups/17/devices/0000:17:0a.7
/sys/kernel/iommu_groups/18/devices/0000:17:0b.0
/sys/kernel/iommu_groups/18/devices/0000:17:0b.1
/sys/kernel/iommu_groups/18/devices/0000:17:0b.2
/sys/kernel/iommu_groups/18/devices/0000:17:0b.3
/sys/kernel/iommu_groups/19/devices/0000:17:0e.0
/sys/kernel/iommu_groups/19/devices/0000:17:0e.1
/sys/kernel/iommu_groups/19/devices/0000:17:0e.2
/sys/kernel/iommu_groups/19/devices/0000:17:0e.3
/sys/kernel/iommu_groups/19/devices/0000:17:0e.4
/sys/kernel/iommu_groups/19/devices/0000:17:0e.5
/sys/kernel/iommu_groups/19/devices/0000:17:0e.6
/sys/kernel/iommu_groups/19/devices/0000:17:0e.7
/sys/kernel/iommu_groups/1/devices/0000:00:04.0
/sys/kernel/iommu_groups/1/devices/0000:00:04.1
/sys/kernel/iommu_groups/1/devices/0000:00:04.2
/sys/kernel/iommu_groups/1/devices/0000:00:04.3
/sys/kernel/iommu_groups/1/devices/0000:00:04.4
/sys/kernel/iommu_groups/1/devices/0000:00:04.5
/sys/kernel/iommu_groups/1/devices/0000:00:04.6
/sys/kernel/iommu_groups/1/devices/0000:00:04.7
/sys/kernel/iommu_groups/20/devices/0000:17:0f.0
/sys/kernel/iommu_groups/20/devices/0000:17:0f.1
/sys/kernel/iommu_groups/20/devices/0000:17:0f.2
/sys/kernel/iommu_groups/20/devices/0000:17:0f.3
/sys/kernel/iommu_groups/20/devices/0000:17:0f.4
/sys/kernel/iommu_groups/20/devices/0000:17:0f.5
/sys/kernel/iommu_groups/20/devices/0000:17:0f.6
/sys/kernel/iommu_groups/20/devices/0000:17:0f.7
/sys/kernel/iommu_groups/21/devices/0000:17:10.0
/sys/kernel/iommu_groups/21/devices/0000:17:10.1
/sys/kernel/iommu_groups/21/devices/0000:17:10.2
/sys/kernel/iommu_groups/21/devices/0000:17:10.3
/sys/kernel/iommu_groups/21/devices/0000:17:10.4
/sys/kernel/iommu_groups/21/devices/0000:17:10.5
/sys/kernel/iommu_groups/21/devices/0000:17:10.6
/sys/kernel/iommu_groups/21/devices/0000:17:10.7
/sys/kernel/iommu_groups/22/devices/0000:17:11.0
/sys/kernel/iommu_groups/22/devices/0000:17:11.1
/sys/kernel/iommu_groups/22/devices/0000:17:11.2
/sys/kernel/iommu_groups/22/devices/0000:17:11.3
/sys/kernel/iommu_groups/23/devices/0000:17:1d.0
/sys/kernel/iommu_groups/23/devices/0000:17:1d.1
/sys/kernel/iommu_groups/23/devices/0000:17:1d.2
/sys/kernel/iommu_groups/23/devices/0000:17:1d.3
/sys/kernel/iommu_groups/24/devices/0000:17:1e.0
/sys/kernel/iommu_groups/24/devices/0000:17:1e.1
/sys/kernel/iommu_groups/24/devices/0000:17:1e.2
/sys/kernel/iommu_groups/24/devices/0000:17:1e.3
/sys/kernel/iommu_groups/24/devices/0000:17:1e.4
/sys/kernel/iommu_groups/24/devices/0000:17:1e.5
/sys/kernel/iommu_groups/24/devices/0000:17:1e.6
/sys/kernel/iommu_groups/25/devices/0000:18:00.0
/sys/kernel/iommu_groups/26/devices/0000:3a:05.0
/sys/kernel/iommu_groups/26/devices/0000:3a:05.2
/sys/kernel/iommu_groups/26/devices/0000:3a:05.4
/sys/kernel/iommu_groups/27/devices/0000:3a:08.0
/sys/kernel/iommu_groups/28/devices/0000:3a:09.0
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.0
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.1
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.2
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.3
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.4
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.5
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.6
/sys/kernel/iommu_groups/29/devices/0000:3a:0a.7
/sys/kernel/iommu_groups/2/devices/0000:00:05.0
/sys/kernel/iommu_groups/2/devices/0000:00:05.2
/sys/kernel/iommu_groups/2/devices/0000:00:05.4
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.0
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.1
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.2
/sys/kernel/iommu_groups/30/devices/0000:3a:0b.3
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.0
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.1
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.2
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.3
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.4
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.5
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.6
/sys/kernel/iommu_groups/31/devices/0000:3a:0c.7
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.0
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.1
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.2
/sys/kernel/iommu_groups/32/devices/0000:3a:0d.3
/sys/kernel/iommu_groups/33/devices/0000:5d:02.0
/sys/kernel/iommu_groups/34/devices/0000:5d:05.0
/sys/kernel/iommu_groups/34/devices/0000:5d:05.2
/sys/kernel/iommu_groups/34/devices/0000:5d:05.4
/sys/kernel/iommu_groups/35/devices/0000:5d:0e.0
/sys/kernel/iommu_groups/35/devices/0000:5d:0e.1
/sys/kernel/iommu_groups/36/devices/0000:5d:0f.0
/sys/kernel/iommu_groups/36/devices/0000:5d:0f.1
/sys/kernel/iommu_groups/37/devices/0000:5d:10.0
/sys/kernel/iommu_groups/37/devices/0000:5d:10.1
/sys/kernel/iommu_groups/38/devices/0000:5d:12.0
/sys/kernel/iommu_groups/38/devices/0000:5d:12.1
/sys/kernel/iommu_groups/38/devices/0000:5d:12.2
/sys/kernel/iommu_groups/38/devices/0000:5d:12.4
/sys/kernel/iommu_groups/38/devices/0000:5d:12.5
/sys/kernel/iommu_groups/39/devices/0000:5d:15.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.1
/sys/kernel/iommu_groups/3/devices/0000:00:08.2
/sys/kernel/iommu_groups/40/devices/0000:5d:16.0
/sys/kernel/iommu_groups/40/devices/0000:5d:16.4
/sys/kernel/iommu_groups/41/devices/0000:5d:17.0
/sys/kernel/iommu_groups/42/devices/0000:5e:00.0
/sys/kernel/iommu_groups/43/devices/0000:5f:03.0
/sys/kernel/iommu_groups/44/devices/0000:60:00.0
/sys/kernel/iommu_groups/45/devices/0000:60:00.1
/sys/kernel/iommu_groups/46/devices/0000:80:04.0
/sys/kernel/iommu_groups/46/devices/0000:80:04.1
/sys/kernel/iommu_groups/46/devices/0000:80:04.2
/sys/kernel/iommu_groups/46/devices/0000:80:04.3
/sys/kernel/iommu_groups/46/devices/0000:80:04.4
/sys/kernel/iommu_groups/46/devices/0000:80:04.5
/sys/kernel/iommu_groups/46/devices/0000:80:04.6
/sys/kernel/iommu_groups/46/devices/0000:80:04.7
/sys/kernel/iommu_groups/47/devices/0000:80:05.0
/sys/kernel/iommu_groups/47/devices/0000:80:05.2
/sys/kernel/iommu_groups/47/devices/0000:80:05.4
/sys/kernel/iommu_groups/48/devices/0000:80:08.0
/sys/kernel/iommu_groups/48/devices/0000:80:08.1
/sys/kernel/iommu_groups/48/devices/0000:80:08.2
/sys/kernel/iommu_groups/49/devices/0000:85:05.0
/sys/kernel/iommu_groups/49/devices/0000:85:05.2
/sys/kernel/iommu_groups/49/devices/0000:85:05.4
/sys/kernel/iommu_groups/4/devices/0000:00:11.0
/sys/kernel/iommu_groups/4/devices/0000:00:11.1
/sys/kernel/iommu_groups/4/devices/0000:00:11.5
/sys/kernel/iommu_groups/50/devices/0000:85:08.0
/sys/kernel/iommu_groups/50/devices/0000:85:08.1
/sys/kernel/iommu_groups/50/devices/0000:85:08.2
/sys/kernel/iommu_groups/50/devices/0000:85:08.3
/sys/kernel/iommu_groups/50/devices/0000:85:08.4
/sys/kernel/iommu_groups/50/devices/0000:85:08.5
/sys/kernel/iommu_groups/50/devices/0000:85:08.6
/sys/kernel/iommu_groups/50/devices/0000:85:08.7
/sys/kernel/iommu_groups/51/devices/0000:85:09.0
/sys/kernel/iommu_groups/51/devices/0000:85:09.1
/sys/kernel/iommu_groups/51/devices/0000:85:09.2
/sys/kernel/iommu_groups/51/devices/0000:85:09.3
/sys/kernel/iommu_groups/51/devices/0000:85:09.4
/sys/kernel/iommu_groups/51/devices/0000:85:09.5
/sys/kernel/iommu_groups/51/devices/0000:85:09.6
/sys/kernel/iommu_groups/51/devices/0000:85:09.7
/sys/kernel/iommu_groups/52/devices/0000:85:0a.0
/sys/kernel/iommu_groups/52/devices/0000:85:0a.1
/sys/kernel/iommu_groups/52/devices/0000:85:0a.2
/sys/kernel/iommu_groups/52/devices/0000:85:0a.3
/sys/kernel/iommu_groups/52/devices/0000:85:0a.4
/sys/kernel/iommu_groups/52/devices/0000:85:0a.5
/sys/kernel/iommu_groups/52/devices/0000:85:0a.6
/sys/kernel/iommu_groups/52/devices/0000:85:0a.7
/sys/kernel/iommu_groups/53/devices/0000:85:0b.0
/sys/kernel/iommu_groups/53/devices/0000:85:0b.1
/sys/kernel/iommu_groups/53/devices/0000:85:0b.2
/sys/kernel/iommu_groups/53/devices/0000:85:0b.3
/sys/kernel/iommu_groups/54/devices/0000:85:0e.0
/sys/kernel/iommu_groups/54/devices/0000:85:0e.1
/sys/kernel/iommu_groups/54/devices/0000:85:0e.2
/sys/kernel/iommu_groups/54/devices/0000:85:0e.3
/sys/kernel/iommu_groups/54/devices/0000:85:0e.4
/sys/kernel/iommu_groups/54/devices/0000:85:0e.5
/sys/kernel/iommu_groups/54/devices/0000:85:0e.6
/sys/kernel/iommu_groups/54/devices/0000:85:0e.7
/sys/kernel/iommu_groups/55/devices/0000:85:0f.0
/sys/kernel/iommu_groups/55/devices/0000:85:0f.1
/sys/kernel/iommu_groups/55/devices/0000:85:0f.2
/sys/kernel/iommu_groups/55/devices/0000:85:0f.3
/sys/kernel/iommu_groups/55/devices/0000:85:0f.4
/sys/kernel/iommu_groups/55/devices/0000:85:0f.5
/sys/kernel/iommu_groups/55/devices/0000:85:0f.6
/sys/kernel/iommu_groups/55/devices/0000:85:0f.7
/sys/kernel/iommu_groups/56/devices/0000:85:10.0
/sys/kernel/iommu_groups/56/devices/0000:85:10.1
/sys/kernel/iommu_groups/56/devices/0000:85:10.2
/sys/kernel/iommu_groups/56/devices/0000:85:10.3
/sys/kernel/iommu_groups/56/devices/0000:85:10.4
/sys/kernel/iommu_groups/56/devices/0000:85:10.5
/sys/kernel/iommu_groups/56/devices/0000:85:10.6
/sys/kernel/iommu_groups/56/devices/0000:85:10.7
/sys/kernel/iommu_groups/57/devices/0000:85:11.0
/sys/kernel/iommu_groups/57/devices/0000:85:11.1
/sys/kernel/iommu_groups/57/devices/0000:85:11.2
/sys/kernel/iommu_groups/57/devices/0000:85:11.3
/sys/kernel/iommu_groups/58/devices/0000:85:1d.0
/sys/kernel/iommu_groups/58/devices/0000:85:1d.1
/sys/kernel/iommu_groups/58/devices/0000:85:1d.2
/sys/kernel/iommu_groups/58/devices/0000:85:1d.3
/sys/kernel/iommu_groups/59/devices/0000:85:1e.0
/sys/kernel/iommu_groups/59/devices/0000:85:1e.1
/sys/kernel/iommu_groups/59/devices/0000:85:1e.2
/sys/kernel/iommu_groups/59/devices/0000:85:1e.3
/sys/kernel/iommu_groups/59/devices/0000:85:1e.4
/sys/kernel/iommu_groups/59/devices/0000:85:1e.5
/sys/kernel/iommu_groups/59/devices/0000:85:1e.6
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/60/devices/0000:ae:05.0
/sys/kernel/iommu_groups/60/devices/0000:ae:05.2
/sys/kernel/iommu_groups/60/devices/0000:ae:05.4
/sys/kernel/iommu_groups/61/devices/0000:ae:08.0
/sys/kernel/iommu_groups/62/devices/0000:ae:09.0
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.0
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.1
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.2
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.3
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.4
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.5
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.6
/sys/kernel/iommu_groups/63/devices/0000:ae:0a.7
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.0
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.1
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.2
/sys/kernel/iommu_groups/64/devices/0000:ae:0b.3
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.0
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.1
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.2
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.3
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.4
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.5
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.6
/sys/kernel/iommu_groups/65/devices/0000:ae:0c.7
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.0
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.1
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.2
/sys/kernel/iommu_groups/66/devices/0000:ae:0d.3
/sys/kernel/iommu_groups/67/devices/0000:d7:05.0
/sys/kernel/iommu_groups/67/devices/0000:d7:05.2
/sys/kernel/iommu_groups/67/devices/0000:d7:05.4
/sys/kernel/iommu_groups/68/devices/0000:d7:0e.0
/sys/kernel/iommu_groups/68/devices/0000:d7:0e.1
/sys/kernel/iommu_groups/69/devices/0000:d7:0f.0
/sys/kernel/iommu_groups/69/devices/0000:d7:0f.1
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:16.1
/sys/kernel/iommu_groups/6/devices/0000:00:16.4
/sys/kernel/iommu_groups/70/devices/0000:d7:10.0
/sys/kernel/iommu_groups/70/devices/0000:d7:10.1
/sys/kernel/iommu_groups/71/devices/0000:d7:12.0
/sys/kernel/iommu_groups/71/devices/0000:d7:12.1
/sys/kernel/iommu_groups/71/devices/0000:d7:12.2
/sys/kernel/iommu_groups/71/devices/0000:d7:12.4
/sys/kernel/iommu_groups/71/devices/0000:d7:12.5
/sys/kernel/iommu_groups/72/devices/0000:d7:15.0
/sys/kernel/iommu_groups/73/devices/0000:d7:16.0
/sys/kernel/iommu_groups/73/devices/0000:d7:16.4
/sys/kernel/iommu_groups/74/devices/0000:d7:17.0
/sys/kernel/iommu_groups/75/devices/0000:60:02.0
/sys/kernel/iommu_groups/76/devices/0000:60:02.1
/sys/kernel/iommu_groups/77/devices/0000:60:02.2
/sys/kernel/iommu_groups/78/devices/0000:60:02.3
/sys/kernel/iommu_groups/79/devices/0000:60:06.0
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
/sys/kernel/iommu_groups/80/devices/0000:60:06.1
/sys/kernel/iommu_groups/81/devices/0000:60:06.2
/sys/kernel/iommu_groups/82/devices/0000:60:06.3
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.5
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
4,074
368
88
31
Vienna
could you both try to set the machine type to 'pc-q35-3.1'

there were some changes in qemu 4 (which should be fixed with 4.0.1 or 4.1, depending what we will package) that can be
fixed with temporarily changing to a 3.1 machine type
 
  • Like
Reactions: BobhWasatch

Ming-Yuan Yu

New Member
Aug 8, 2019
3
0
1
28
could you both try to set the machine type to 'pc-q35-3.1'

there were some changes in qemu 4 (which should be fixed with 4.0.1 or 4.1, depending what we will package) that can be
fixed with temporarily changing to a 3.1 machine type
Setting machine type to `pc-q35-3.1` does not seem to change the behavior. It still freezes at `Guest has not initialized the display (yet)` when setting `pcie=1` on other VFs.

I forget to mention that my NICs are Intel X722.
 

cryse

New Member
Nov 30, 2019
1
0
1
29
I have the same issue, cannot start second VM if pcie=1 enabled.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
4,074
368
88
31
Vienna
could you try the newer kernel from the repos (5.3)?
maybe it is a driver issue...

i have here an I350 with sr-iov, and i can passthrough multiple vfs in either pci/pcie mode without problems....
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!