Iommu ID persistence

[HellrazorX]

Hi all,

Very sorry if the topic is already adressed (pun intended)
I read the documentation of Iommu and this behaviour doesn't seems to be covered (or incompatible jargon)

Is there a way to ensure that IOMMU device adress shifts along with hardware changes?
IOMMU id's seems to be arbitrairy indexed from 1 to X to whatever PCIe devices availlable.
If a IOMMU id 5 is given to a VM, hardware changes could make this ID shift to 6 and now, another pci device will be passed to the vm.

Let me show two examples to make sure where I'm going to.

-Got a LSI HBA passthrough for one of my vms. ID 5
-If I add one NIC inside the system and boot proxmox. This nic is now the device passed through instead. Because Nic is now ID5 and ID6 is now my HBA.

This previous case is fairly easy to recover from. Simply select the proper PCI device within the GUI and launch the VM again.

THis next case is the reason why I'm posting the question.

-I got a NIC passthrough a pfsense VM.
-Now I know that whenever I'll make a hardware change, if will lose connectivity the server if the PCIe order changes.

The only way to recover from that is to physically connect to the other Lan port to correct the wrong adressing and network will be back.

---

I know it's possible to make a correpondance table with SATA ports passthrough with HDD Models serials etc to keep persistence. I can't help but imagine that the issue I'm observing with PCI passthrought is known / solved.

Thanks a lot for pointing me the right direction.

HX

 

[leesteken]

Allow me to make a distinction between PCI ID and IOMMU group ID. Every PCI(e) device in the system has a PCI ID (something like 0000:0a:00.0), and it is in an IOMMU group with a number (like 5). Although the IOMMU group is imporant (because you cannot share devices in a group between VMs and the Proxmox host), the PCI ID is used to configure the passthrough. Therefore, IOMMU group number is irrelevant and there should be no problem when it changes.

The way most motherboard assign PCI IDs to PCI(e) devices does however often change when adding additional devices via PCIe slots and M.2 slots, or even when enabling or disabling devices in the BIOS. With modern Linux, ethernet devices have a name based on the PCI ID (instead of eth1, which would often change between reboots). Therefore, currently, any change to the number of PCI(e) devices can break passthrough because the IDs of the various devices can shift up or down. This can cause issues from VMs failing to start, to not having a network connection anymore, to Proxmox host crashes and reboots.

People run into this every once in a while, but I don't know if it is fixable as long as motherboards keep shifting the PCI IDs. Even rewriting QEMU/KVM/VFIO to use the numeric device ID (like 8086:1112) won't fix the problem if you have multiple devices of a certain brand and type.

 

[HellrazorX]

Allow me to make a distinction between PCI ID and IOMMU group ID. Every PCI(e) device in the system has a PCI ID (something like 0000:0a:00.0), and it is in an IOMMU group with a number (like 5). Although the IOMMU group is imporant (because you cannot share devices in a group between VMs and the Proxmox host), the PCI ID is used to configure the passthrough. Therefore, IOMMU group number is irrelevant and there should be no problem when it changes.

The way most motherboard assign PCI IDs to PCI(e) devices does however often change when adding additional devices via PCIe slots and M.2 slots, or even when enabling or disabling devices in the BIOS. With modern Linux, ethernet devices have a name based on the PCI ID (instead of eth1, which would often change between reboots). Therefore, currently, any change to the number of PCI(e) devices can break passthrough because the IDs of the various devices can shift up or down. This can cause issues from VMs failing to start, to not having a network connection anymore, to Proxmox host crashes and reboots.

People run into this every once in a while, but I don't know if it is fixable as long as motherboards keep shifting the PCI IDs. Even rewriting QEMU/KVM/VFIO to use the numeric device ID (like 8086:1112) won't fix the problem if you have multiple devices of a certain brand and type.

I understand how complex/impossible to take into account All bios behaviors accross space and time,

The way I read you, there are no way to determine which Serial number was associated to a particular IOMMU group and remap the newly numbered group to the VM(s) it was associated to.

Also I don't know if a PCIE 1x lane can be Grouped to another onboard device which can Cause a complete crash of the system. Supposing the device, associated to a group on which it was alone on it's lane and was displaced to this lane that is shared for Ie: Shared with Some sata ctrl.

Thanks for your explanation