PCI Passthrough, iommu, ACS and isolation

[tld_it]

Hello,
I'm new to ProxMox, trying out PVE8. This is a first time post on a forum, so excuse me if I'm not following good practices (please inform me if I do so).
Also no professional networking knowledge and new to Linux.

I've been working on my small home server, installing a quad-port nic, with intention to install virtualized OPNsense.

Passing through the nic seems to have worked, using the tips in several threads and tutorials in this forum.
Activated intel iommu in cmdline, added the necessary vfio entries in the modules section, and managed to activate the vfio-pci driver on the nic, over the standard igb network driver using a .conf file in the modprobe.d directory with options vfio-pci ids (vendor) : (device) and softdep command to override loading of the standard driver. So far so good. And I learned a lot.

The question that keeps running through my mind, and that i could not find a clear answer to, is the possible risk of a VM-breakout due to poor isolation of data traffic on the nic being passed through, in a specific situation. Mine.
So I'm running on a Xeon E3 1226, C220 chipset, and installed a NC365T nic (HP). This network interface is based on the intel i340 (82580) chipset. This is not SR-IOV capable. In the devices list, the four ports all appear with same vendor and device id (therefor all four ports are passed through with vfio-pci driver.

04:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
        Subsystem: Hewlett-Packard Company NC365T 4-port Ethernet Server Adapter [103c:1780]
        Kernel driver in use: vfio-pci
        Kernel modules: igb
04:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
        Subsystem: Hewlett-Packard Company NC365T 4-port Ethernet Server Adapter [103c:1780]
        Kernel driver in use: vfio-pci
        Kernel modules: igb
04:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
        Subsystem: Hewlett-Packard Company NC365T 4-port Ethernet Server Adapter [103c:1780]
        Kernel driver in use: vfio-pci
        Kernel modules: igb
04:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
        Subsystem: Hewlett-Packard Company NC365T 4-port Ethernet Server Adapter [103c:1780]
        Kernel driver in use: vfio-pci
        Kernel modules: igb

Though, as of installation, all four individual ports appear to be in their own IOMMU group.
I did not use the acs override patch, as I'm aware that is a risk and there's no need for it as the interfaces are in their own group.

┌──────────┬────────┬──────────────┬────────────┬────────┬────────────────────────────────────────────────────────────────────────── │ class │ device │ id │ iommugroup │ vendor │ device_name ╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════ │ 0x010601 │ 0x8c02 │ 0000:00:1f.2 │ 12 │ 0x8086 │ 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x020000 │ 0x153a │ 0000:00:19.0 │ 5 │ 0x8086 │ Ethernet Connection I217-LM ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x020000 │ 0x150e │ 0000:04:00.0 │ 14 │ 0x8086 │ 82580 Gigabit Network Connection ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x020000 │ 0x150e │ 0000:04:00.1 │ 15 │ 0x8086 │ 82580 Gigabit Network Connection ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x020000 │ 0x150e │ 0000:04:00.2 │ 16 │ 0x8086 │ 82580 Gigabit Network Connection ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x020000 │ 0x150e │ 0000:04:00.3 │ 17 │ 0x8086 │ 82580 Gigabit Network Connection ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x030000 │ 0x041a │ 0000:00:02.0 │ 0 │ 0x8086 │ Xeon E3-1200 v3 Processor Integrated Graphics Controller ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x040300 │ 0x0c0c │ 0000:00:03.0 │ 2 │ 0x8086 │ Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x040300 │ 0x8c20 │ 0000:00:1b.0 │ 7 │ 0x8086 │ 8 Series/C220 Series Chipset High Definition Audio Controller ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060000 │ 0x0c08 │ 0000:00:00.0 │ 1 │ 0x8086 │ Xeon E3-1200 v3 Processor DRAM Controller ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060100 │ 0x8c56 │ 0000:00:1f.0 │ 12 │ 0x8086 │ C226 Series Chipset Family Server Advanced SKU LPC Controller ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060400 │ 0x8c10 │ 0000:00:1c.0 │ 8 │ 0x8086 │ 8 Series/C220 Series Chipset Family PCI Express Root Port #1 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060400 │ 0x8c16 │ 0000:00:1c.3 │ 9 │ 0x8086 │ 8 Series/C220 Series Chipset Family PCI Express Root Port #4 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060400 │ 0x8c18 │ 0000:00:1c.4 │ 10 │ 0x8086 │ 8 Series/C220 Series Chipset Family PCI Express Root Port #5 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x060401 │ 0x8893 │ 0000:02:00.0 │ 13 │ 0x1283 │ IT8893E PCIe to PCI Bridge ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x078000 │ 0x8c3a │ 0000:00:16.0 │ 4 │ 0x8086 │ 8 Series/C220 Series Chipset Family MEI Controller #1 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x0c0320 │ 0x8c2d │ 0000:00:1a.0 │ 6 │ 0x8086 │ 8 Series/C220 Series Chipset Family USB EHCI #2 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x0c0320 │ 0x8c26 │ 0000:00:1d.0 │ 11 │ 0x8086 │ 8 Series/C220 Series Chipset Family USB EHCI #1 ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x0c0330 │ 0x8c31 │ 0000:00:14.0 │ 3 │ 0x8086 │ 8 Series/C220 Series Chipset Family USB xHCI ├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────────────────────────── │ 0x0c0500 │ 0x8c22 │ 0000:00:1f.3 │ 12 │ 0x8086 │ 8 Series/C220 Series Chipset Family SMBus Controller └──────────┴────────┴──────────────┴────────────┴────────┴──────────────────────────────────────────────────────────────────────────

Now the Xeon E3-1200 series appear to be known for their lack of ACS capability, with Intel recommending not to use passing-through of devices to VMs.

I can't find an answer to the question: is this configuration at risk, especially if one of the ports of the nic is facing internet.
Is a secure separation of traffic in place because of the split iommu groups, or do the iommu groups only appear to be separated because of the CPU not being ACS capable (verbose output of lspci lists no ACSCap on any device) like one would have used the acs-override patch.

Many thanks,
Jo

 

[leesteken]

In principle, PCI(e) devices that can communicate between themselves, without the CPU/IOMMU noticing it, are put in the same IOMMU group. Therefore, you should be fine passing through the four functions of the network controller to separate VMs and/or the Proxmox host.
The pcie_acs_overrride breaks exactly this isolation, which would allow one device in a group to read all (VM or host) memory and communicate it to another VM secretly (but most people don't seem to care).

Now the Xeon E3-1200 series appear to be known for their lack of ACS capability, with Intel recommending not to use passing-through of devices to VMs.

The IOMMU grouping, where everything is in a separate group, does look too good to be true (but it could happen, on a server system). Do you have a dependable source for this?
If it's true then the Linux kernel should put everything into a single IOMMU group. Maybe you have a newer revision without the issue? Maybe it's a bug in the kernel? Maybe there is a BIOS setting that controls this?

 

[tld_it]

Good evening,

Thanks for the fast reply.

The IOMMU grouping, where everything is in a separate group, does look too good to be true (but it could happen, on a server system). Do you have a dependable source for this?

I'm using a Lenovo is8xm Sharkbay board from a Thinkstation p300, which is a server board (also using ecc-ram).

tld_it said:
Now the Xeon E3-1200 series appear to be known for their lack of ACS capability, with Intel recommending not to use passing-through of devices to VMs.

If it's true then the Linux kernel should put everything into a single IOMMU group. Maybe you have a newer revision without the issue? Maybe it's a bug in the kernel? Maybe there is a BIOS setting that controls this?

The processor is a 1226 v3, but no indication of added ACS. I'm on the most recent BIOS (2022) and can't recall a BIOS setting specific for ACS (as most motherboards do), just enable or disable vt-d as far as I recall. I would have to double check tomorrow.

Dank u,
Jo

 

[tld_it]

I did double check the BIOS settings of the server, and like I thought, the only settings available for the CPU regarding virtualization is vt-x and vt-d technology (option enable/disable).
BIOS version is 23/12/2021.

 

[tld_it]

I found a document from Lenovo on iommu in Linux, although it doesn't really answer my specific question, it's quite detailed.

An Introduction to IOMMU Infrastructure in the Linux Kernel

 

[leesteken]

I did double check the BIOS settings of the server, and like I thought, the only settings available for the CPU regarding virtualization is vt-x and vt-d technology (option enable/disable).
BIOS version is 23/12/2021.

The lack of settings in the BIOS does not mean functionality does not exist. It only means that the creator of the BIOS did not want to give you the option to enable or disable it.
Can you point to some concrete information that confirms that the E3-1200 series does not properly isolate passed through devices? Currently, it looks like ACS is automatically enabled.

 

[tld_it]

What I read in this document from Intel on this processor family doesn't seem positive to me.
See page 64 (last page).
Intel® Xeon® E3-1200 v3 Processor Family
It points specific at PCIe root port. Does that make any difference?