[SOLVED] PCIE Passthrough to Specific VM

[Nollimox]

So, IOMMU is enabled and I have two PCIE devices that I want to pas Is-through to a specific VM; however, when I add hardware, PCI device, one doesn't show and the other, it seems that I need to add each port instead of the entire device...why? Please see below...
In can input the device vendor and ID but the drop down doesn't have one of the device.
Computer is a Dell Precision 5820 Xeon w-2155 which must add PCIe Graphics card

root@nolliprivatecloud:~# dmesg | grep -e IOMMU
[ 0.170223] DMAR: IOMMU enabled
[ 0.450310] DMAR-IR: IOAPIC id 12 under DRHD base 0xfbffc000 IOMMU 2
[ 0.450311] DMAR-IR: IOAPIC id 11 under DRHD base 0xd8ffc000 IOMMU 1
[ 0.450312] DMAR-IR: IOAPIC id 10 under DRHD base 0xb5ffc000 IOMMU 0
[ 0.450313] DMAR-IR: IOAPIC id 8 under DRHD base 0x92ffc000 IOMMU 3
[ 0.450314] DMAR-IR: IOAPIC id 9 under DRHD base 0x92ffc000 IOMMU 3
root@nolliprivatecloud
GNU nano 7.2 /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
option vfio-pci ids=8086:1521, 8086:0435
vfio_virqfd

 

[Nollimox]

root@nolliprivatecloud:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 0000:bc:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
IOMMU group 10 0000:00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 11 0000:00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 12 0000:00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 13 0000:00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 14 0000:00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 15 0000:00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 16 0000:00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU group 17 0000:00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
IOMMU group 18 0000:00:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS [8086:2025] (rev 04)
IOMMU group 19 0000:00:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOAPIC [8086:2026] (rev 04)
IOMMU group 1 0000:bc:05.5 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller [8086:201d] (rev 04)
IOMMU group 1 10000:00:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 04)
IOMMU group 1 10000:00:03.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port D [8086:2033] (rev 04)
IOMMU group 1 10000:01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU group 1 10000:02:00.0 Non-Volatile memory controller [0108]: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1602 [1e4b:1602] (rev 01)
IOMMU group 20 0000:00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
IOMMU group 21 0000:00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
IOMMU group 22 0000:00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
IOMMU group 23 0000:00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
IOMMU group 23 0000:00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
IOMMU group 24 0000:00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU group 25 0000:00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU group 26 0000:00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
IOMMU group 27 0000:00:1c.5 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #6 [8086:a295] (rev f0)
IOMMU group 28 0000:00:1c.6 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #7 [8086:a296] (rev f0)
IOMMU group 29 0000:00:1f.0 ISA bridge [0601]: Intel Corporation C422 Chipset LPC/eSPI Controller [8086:a2d3]
IOMMU group 29 0000:00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
IOMMU group 29 0000:00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU group 29 0000:00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU group 29 0000:00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3]
IOMMU group 2 0000:bd:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [NVS 510] [10de:0ffd] (rev a1)
IOMMU group 2 0000:bd:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
IOMMU group 30 0000:02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 31 0000:03:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 32 0000:03:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
IOMMU group 33 0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
IOMMU group 34 0000:06:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
IOMMU group 35 0000:4e:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
IOMMU group 36 0000:4e:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU group 37 0000:4e:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04)
IOMMU group 38 0000:4e:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 38 0000:4e:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 39 0000:4e:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 39 0000:4e:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU group 3 0000:90:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
IOMMU group 40 0000:4e:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 40 0000:4e:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 41 0000:4e:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 41 0000:4e:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU group 42 0000:4e:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
IOMMU group 42 0000:4e:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
IOMMU group 42 0000:4e:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
IOMMU group 42 0000:4e:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
IOMMU group 43 0000:4e:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
IOMMU group 43 0000:4e:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
IOMMU group 43 0000:4e:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
IOMMU group 43 0000:4e:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
IOMMU group 43 0000:4e:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
IOMMU group 43 0000:4e:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
IOMMU group 43 0000:4e:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
IOMMU group 44 0000:90:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
IOMMU group 45 0000:90:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU group 46 0000:90:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04)
IOMMU group 47 0000:90:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 04)
IOMMU group 48 0000:90:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 04)
IOMMU group 49 0000:90:0a.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 04)
IOMMU group 4 0000:91:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 50 0000:90:0a.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 04)
IOMMU group 51 0000:90:0a.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 04)
IOMMU group 52 0000:90:0a.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 04)
IOMMU group 53 0000:90:0a.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 04)
IOMMU group 54 0000:90:0a.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 04)
IOMMU group 55 0000:90:0a.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 04)
IOMMU group 56 0000:90:0a.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
IOMMU group 57 0000:90:0b.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
IOMMU group 58 0000:90:0b.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
IOMMU group 59 0000:90:0b.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
IOMMU group 5 0000:91:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 60 0000:90:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
IOMMU group 61 0000:90:0c.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 04)
IOMMU group 62 0000:90:0c.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 04)
IOMMU group 63 0000:90:0c.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 04)
IOMMU group 64 0000:90:0c.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 04)
IOMMU group 65 0000:90:0c.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 04)
IOMMU group 66 0000:90:0c.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 04)
IOMMU group 67 0000:90:0c.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 04)
IOMMU group 68 0000:90:0c.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 04)
IOMMU group 69 0000:90:0d.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 04)
IOMMU group 6 0000:91:00.2 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 70 0000:90:0d.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 04)
IOMMU group 71 0000:90:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 04)
IOMMU group 72 0000:90:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 04)
IOMMU group 73 0000:bc:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 04)
IOMMU group 74 0000:bc:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU group 75 0000:bc:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 04)
IOMMU group 76 0000:bc:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
IOMMU group 77 0000:bc:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
IOMMU group 77 0000:bc:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
IOMMU group 78 0000:bc:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU group 79 0000:bc:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU group 79 0000:bc:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU group 7 0000:91:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
IOMMU group 80 0000:bc:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU group 8 0000:00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
IOMMU group 9 0000:00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
root@nolliprivatecloud:~#

The two Items in bold are what I want to passthrough to a specific VM...how to do when it doesn't show under hardware?

 

[leesteken]

Are you sure that you are not using pcie_acs_override, which invalidates the IOMMU group information and breaks the secure device isolation?

The two Items in bold are what I want to passthrough to a specific VM...how to do when it doesn't show under hardware?

You edit the VM configuration file, in the /etc/pve/qemu-server/ directory, manually using nano or any other text editor you prefer.

GNU nano 7.2 /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
option vfio-pci ids=8086:1521, 8086:0435
vfio_virqfd

Note that vfio_virqfd is no longer a separate module in PVE 8. Also, option vfio-pci ids=8086:1521, 8086:0435 should not go in /etc/modules, where it is ignored as it should only contains module names. Put it in a .conf file in the /etc/modprobe.d/ directory instead.

 

[Nollimox]

Are you sure that you are not using pcie_acs_override, which invalidates the IOMMU group information and breaks the secure device isolation?

You edit the VM configuration file, in the /etc/pve/qemu-server/ directory, manually using nano or any other text editor you prefer.

Note that vfio_virqfd is no longer a separate module in PVE 8. Also, option vfio-pci ids=8086:1521, 8086:0435 should not go in /etc/modules, where it is ignored as it should only contains module names. Put it in a .conf file in the /etc/modprobe.d/ directory instead.

Thank you Leesteken, was hoping that you responded. So, I'll add it to the blacklist.conf file in /etc/modprobe.d then add to VM configuration file...really appreciate it.

 

[Nollimox]

Done deal...thank you!

root@nolliprivatecloud:~# qm set 100 -hostpci1 0000:91:00.0
update VM 100: -hostpci1 0000:91:00.0
root@nolliprivatecloud:~# qm set 100 -hostpci2 0000:04:00.0
update VM 100: -hostpci2 0000:04:00.0
root@nolliprivatecloud:~#

GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci1: 0000:91:00.0
hostpci2: 0000:04:00.0
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1,firewall=1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae