SR-IOV IOMMU Groups Keep Changing

[kayson]

I have 4 PVE nodes (Lenovo M920q's) with Intel X710-DA2's. I want to pass through one of the SR-IOV VFs to pfsense so it can handle VLANs, etc. Unfortunately, on reboots, the IOMMU groups seem to change. It's not consistent, either, and I'm not sure what the trigger is. After a ton of trial and error, research, and LLM chats, I've settled on the following kernel command line options: quiet intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses mitigations=off video=1280x1024 pcie_acs_override=downstream,multifunction i915.enable_gvt=1 driver_async_probe=off

This works generally, except for the mentioned issue. I tried forcing the link speed and added the last entry (driver async probe) in the hopes that it was some timing issue, but no luck. Is there any way to force a particular group for a particular pci path?

 

[leesteken]

The IOMMU groups are usually determined by the motherboard (physical PCIe layout and BIOS/UEFI). Since you already ignore the security borders put up by IOMMU groups (by using the pcie_acs_override): instead of using Mapped Resources (that check IOMMU group number and device ID), maybe as a work-around (for your motherboard/BIOS) use raw devices in the VM configs (that don't check IOMMU groups and device IDs)?

 

[kayson]

The IOMMU groups are usually determined by the motherboard (physical PCIe layout and BIOS/UEFI). Since you already ignore the security borders put up by IOMMU groups (by using the pcie_acs_override): instead of using Mapped Resources (that check IOMMU group number and device ID), maybe as a work-around (for your motherboard/BIOS) use raw devices in the VM configs (that don't check IOMMU groups and device IDs)?

Unfortunately i need the override otherwise all the VFs go in one group. It's consumer hardware, so the CPU root port doesn't have ACS. With raw devices, I don't think I can migrate VMs between nodes, right?

 

[leesteken]

Unfortunately i need the override otherwise all the VFs go in one group. It's consumer hardware, so the CPU root port doesn't have ACS.

Sounds like it does have ACS and the devices in the group can talk to each other bypassing the IOMMU. That's why you need to ignore the grouping and what makes it a security risk.

With raw devices, I don't think I can migrate VMs between nodes, right?

Probably, unless you happen to use the same hardware/PCI-IDs everywhere. Do you need offline migration a lot (as live migration is not possible anyway)?

 

[kayson]

Turns out the issue was a race condition! I have systemd-udevd link files for each PF that creates the VFs. systemd's rules/events fire in parallel, so whichever VFs happened to get created first got the lower iommu groups. The solution was to add my own udev rules to make the second PF wait for VFs to be created on the first one before running its own VF creation

 

[leesteken]

Thank you for letting us know that this behavior did not come from the hardware. I have no experience with creating Virtual Functions on the fly, so I did not think of this possibly being the case.