IOMMU groups in PCIe Passthrough

[kevindd992002]

Here's the IOMMU groups in my proxmox server:

IOMMU Group 0:
        00:02.0 VGA compatible controller [0300]: Intel Corporation CometLake-S GT2 [UHD Graphics 630] [8086:9bc8] (rev 03)
IOMMU Group 1:
        00:00.0 Host bridge [0600]: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers [8086:9b63] (rev 03)
IOMMU Group 2:
        00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 03)
        01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU Group 3:
        00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
        00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
IOMMU Group 4:
        00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
IOMMU Group 5:
        00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
IOMMU Group 6:
        00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:43bd] (rev 11)
IOMMU Group 7:
        00:1d.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCI Express Root Port #9 [8086:43b0] (rev 11)
IOMMU Group 8:
        00:1f.0 ISA bridge [0601]: Intel Corporation H570 LPC/eSPI Controller [8086:4386] (rev 11)
        00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
        00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (14) I219-V [8086:15fa] (rev 11)
IOMMU Group 9:
        02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 10:
        03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]

I'm trying to passthrough the Intel I219-V NIC to my opnsense VM but I'm not sure if it's possible because it is in the same group as the SMBus and SPI controllers. Why is that the case anyway? Aren't they unrelated at all? What exactly are SPI and SMBus controllers?

 

[leesteken]

I'm trying to passthrough the Intel I219-V NIC to my opnsense VM but I'm not sure if it's possible because it is in the same group as the SMBus and SPI controllers. Why is that the case anyway?

That depends on your motherboard (chipset, PCIe multiplexer chips, BIOS and the IOMMU of the CPU).

Aren't they unrelated at all? What exactly are SPI and SMBus controllers?

According to the IOMMU groups, they can communicate with each other without the IOMMU knowing about it or capable of preventing it.
Therefore putting on of the devices in a VM and keeping the others on the Proxmox host will (because of DMA) allow the VM to read and write all of the Proxmox host memory.
The IOMMU groups are there to make sure the security isolation between memory domains (VMs and/or the host). Maybe search and read up on IOMMU groups?

Try putting the network card in another PCIe slot, as that is the only secure way to change the IOMMU group (as it is put somewhere else in the PCIe layout, which can change the PCI ID of the device and other devices). Usually the PCIe and M.2 slots connected directly to the CPU are in their own groups.

 

[kevindd992002]

Right. But if I dont' really care about security (since this is just a home lab server), can I passthrough just the Intel NIC function to the VM (with ACS override) and call it a day? What I'm reading about IOMMU is that if one function in a certain group is passthrough'ed to a VM, the other functions cannot be used by other VM's or by the host itself. I did read about IOMMU groups but am still confused which is why I resorted to posting here.

The network card is an onboard one, so no moving. I have a consumer ITX board that only has one PCIe slot and that slot is taken already by my HBA card.

 

[kevindd992002]

Or maybe I need to know what SPI and SMBus controllers really do? If SMBus is the controller for USB, then I can go away passing through that to the VM as well. Now I just need to know what SPI is.

 

[leesteken]

Or maybe I need to know what SPI and SMBus controllers really do? If SMBus is the controller for USB, then I can go away passing through that to the VM as well.

It is not a USB controller. Maybe just try it and see what breaks?

 

[kevindd992002]

I guess I can do that. I'm just worried that I break pve or something.

Worse comes to worst, is it really worth it passing through a 1Gb NIC to opnsense (with the security risk of overriding ACS) or just get away with a bridge with a virtio driver in the VM?

 

[leesteken]

I guess I can do that. I'm just worried that I break pve or something.

Don't make it permanent. Just select the (raw) device for passthrough in the VM and don't make it start automatically.

Worse comes to worst, is it really worth it passing through a 1Gb NIC to opnsense (with the security risk of overriding ACS) or just get away with a bridge with a virtio driver in the VM?

Maybe the network controller does not even work with passthrough. Maybe the override does not split the group. Maybe the group is split but it still does not work.
Just test with VirtIO network first and see if it is good enough.

 

[leesteken]

Be aware that the network device is the connection with the outside world (even if it is behind other firewalls etc.). If somebody breaks into the OPNsense VM via the network controller (when using the ACS override) they have access to your whole Proxmox host (and all VMs)!

 

[kevindd992002]

I tested by passing through the NIC and it does look like opnsense sees it. I just don't know if I'm comfortable leaving the three controllers blocked from the host. Here are the kernel drivers for the devices in that IOMMU group:

00:1f.0 ISA bridge: Intel Corporation H570 LPC/eSPI Controller (rev 11)
        Subsystem: ASRock Incorporation H570 LPC/eSPI Controller
00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11)
        Subsystem: ASRock Incorporation Tiger Lake-H SMBus Controller
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
00:1f.5 Serial bus controller: Intel Corporation Tiger Lake-H SPI Controller (rev 11)
        Subsystem: ASRock Incorporation Tiger Lake-H SPI Controller
        Kernel driver in use: intel-spi
        Kernel modules: spi_intel_pci
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (14) I219-V (rev 11)
        Subsystem: ASRock Incorporation Ethernet Connection (14) I219-V
        Kernel driver in use: e1000e
        Kernel modules: e1000e

Doing a quick research:

i801_smbus -> has something to do with low-speed communication between devices in a mobo. It affects sensors (temperature, fan controllers, etc.)
intel-spi -> update/dump BIOS from within linux
LPC/eSPI controller -> not sure why there aren't any drivers loaded for this one but here's what it does: https://www.totalphase.com/blog/2021/09/what-is-the-espi-protocol-and-how-does-it-improve-upon-lpc/

Reading all those, I don't think I want them blocked off. And I'm also hesitant with the whole ACS override thing. I'm going to try virtio for now.

As for HDD passthrough, I have 3 connected to my HBA card and 3 to the onboard SATA ports. I see that the SATA controller and the HBA PCIe card are on their own IOMMU groups. So is passing them through those raw PCI(e) devices to a NAS VM the correct way to go about "direct access" to the disks? Or is this the right way?