SATA controller (PCIe passthrough) and NIC on the same IOMMU group

[alessio1996]

Hello,

I've dived into many discussions/threads/workarounds before writing this thread that I hope it will better clarify my situation.
I'm currently running Proxmox on an intel based system with a Intel Xeon E-2174G CPU.

Currently, I have passed through the PCIe SATA controller to a NAS (truenas) VM (device 02.00.0).
I've recently decided to upgrade my home network to 2.5G so in my case, adding a 2.5G NIC (01.00.0) to the other PCIe available slot (my MB has only 2 of them).

The new NIC is not recognized by Proxmox (maybe) because as you can see, the SATA controller and the NIC appear to be in the same IOMMU group. The NIC should be controlled entirely by the host as it can be shared between all the VMs/LCXs .

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Workstation Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e18] (rev 07)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU Group 1 01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU Group 1 02:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1166 Serial ATA Controller [1b21:1166] (rev 02)
IOMMU Group 2 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU Group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 5 00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 [8086:a368] (rev 10)
IOMMU Group 5 00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 [8086:a369] (rev 10)
IOMMU Group 6 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 6 00:16.4 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller #2 [8086:a364] (rev 10)
IOMMU Group 7 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU Group 8 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU Group 9 00:1b.5 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 [8086:a32d] (rev f0)
IOMMU Group 10 00:1b.6 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 [8086:a32e] (rev f0)
IOMMU Group 11 00:1b.7 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #24 [8086:a32f] (rev f0)
IOMMU Group 12 00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 [8086:a338] (rev f0)
IOMMU Group 13 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 [8086:a330] (rev f0)
IOMMU Group 14 00:1e.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller [8086:a328] (rev 10)
IOMMU Group 15 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a30a] (rev 10)
IOMMU Group 15 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU Group 15 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 16 03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 17 04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 18 05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 19 06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 20 07:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU Group 20 08:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU Group 21 09:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

I've already tried to add to the kernel parameters pcie_acs_override=downstream or pcie_acs_override=downstream,multifunction but without getting the two devices splitting in two different IOMMU groups (moreover, I'm not sure that my CPU supports ACL).

Is there a way to get around on this?

(I can instead bond all 4 in-board NICs to get a 4G bandwidth but you now, it's not a clean solution).


Thank you!
Alessio

 

[leesteken]

Is there a way to get around on this?

The IOMMU groups are determined by the motherboard physical PCIe layout and (PCIe multiplexer) chips and its BIOS. Try different PCIe slots (and maybe M.2 slots via a converter) or buy another motherboard that is known to have better groups.