PCIe Passthrough woes

[kratos-nuc]

I was running ESXi until yesterday, and decided to hop over to Proxmox. I am setting up VMs one by one, and the first issue I ran into was trying to pass through an SSD controller (03:00 in the output below) to OpenMediaVault. The VM would fail to start after spinning for a while with a generic unable to tail, received 0 bytes error. I passed the disk through the host and it works fine.

Now to the main issue, I am trying to pass through an NVIDIA T1000 GPU to an Ubuntu VM. Unfortunately I found out that it is in the same IOMMU group as the boot disk for Proxmox (ext4), and trying to boot the VM with it attached takes down the host (and interestingly switches the ethernet port's enp naming after force reboot even though it isn't in the same IOMMU group). But, I was able to PCI pass through the GPU in ESXi (ESXi was installed in the same disk Proxmox is installed now). Is it not possible to do that in Proxmox? Am I missing something?

PS: I did not understand the implications of applying the ACS patch, and I am not sure if that would help me since I have IOMMU groups.
FWIW I have not disabled/blacklisted the nouveau driver (so lspci shows it being used), but I also made sure that Proxmox uses the integrated graphics for output.

Group 2 being the one containing the GPU and the boot disk.

root@kratos-proxmox:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630] [8086:3e9b]
IOMMU group 10 00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU group 11 00:1c.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #5 [8086:a33c] (rev f0)
IOMMU group 12 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #14 [8086:a335] (rev f0)
IOMMU group 13 00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10)
IOMMU group 14 00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Lake LPC Controller [8086:a30e] (rev 10)
IOMMU group 14 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU group 14 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 14 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 14 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
IOMMU group 15 03:00.0 Non-Volatile memory controller [0108]: Silicon Motion, Inc. SM2262/SM2262EN SSD Controller [126f:2262] (rev 03)
IOMMU group 16 04:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU group 17 05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 18 06:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 19 06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec4] (rev 07)
IOMMU group 20 06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 21 06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
IOMMU group 22 07:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)
IOMMU group 23 3b:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)
IOMMU group 24 70:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 2 00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x4) [8086:1909] (rev 07)
IOMMU group 2 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117GLM [Quadro T400 Mobile] [10de:1fb2] (rev a1)
IOMMU group 2 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)
IOMMU group 2 02:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P2 NVMe PCIe SSD [c0a9:540a] (rev 01)
IOMMU group 3 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU group 4 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU group 5 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 5 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 6 00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10)
IOMMU group 6 00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 [8086:a369] (rev 10)
IOMMU group 7 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU group 8 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake Mobile PCH SATA AHCI Controller [8086:a353] (rev 10)
IOMMU group 9 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)

 

[leesteken]

Now to the main issue, I am trying to pass through an NVIDIA T1000 GPU to an Ubuntu VM. Unfortunately I found out that it is in the same IOMMU group as the boot disk for Proxmox (ext4), and trying to boot the VM with it attached takes down the host (and interestingly switches the ethernet port's enp naming after force reboot even though it isn't in the same IOMMU group). But, I was able to PCI pass through the GPU in ESXi (ESXi was installed in the same disk Proxmox is installed now). Is it not possible to do that in Proxmox? Am I missing something?

PS: I did not understand the implications of applying the ACS patch, and I am not sure if that would help me since I have IOMMU groups.

I assume ESXi ignores the IOMMU groups. You can ignore the IOMMU groups in Proxmox VE by addding pcie_acs_override=downstream,multifunction to the kernel parameters. However, this breaks the security isolation of VMs and I advise against it. The recommended way is to use another PCIe slot that is in another group (and use a motherboard and CPU with good groupings).

IOMMU groups are determined by the motherboard PCIe layout/chips and it's BIOS/UEFI. Devices in the same group can read all of the (VM) memory and communicate with each other without the CPU or IOMMU noticing. A VM can therefore potentially read all memory from another VM and/or the Proxmox host (when some devices not passed through). In case of the host, it also included all memory from all VMs. Use work-arounds at your own risk.

 

[kratos-nuc]

Thank you for the explanation, it is interesting to know PCIe devices in the same IOMMU group can talk to each other directly!

The hardware is an Intel NUC 9i7QNX, which looks like it has separate groups for anything on the NUC board, but the daughterboard with an additional SSD slot and the PCIe slots are sharing a single group. I moved my boot disk to inside the board and I was able to share the GPU. Thanks!