PCIE Passthrough Iommu enumeration changes as devices enabled/disabled

[das1996]

Hopefully the subject was sufficient to grab your attention

Issue: If I enable/disable onboard devices in the bios, iommu groups change, resulting in devices improperly assigned.

System specs:
Asus x570-e motherboard
Amd 5800X processor
64GB Ram
GT710 video card

Onboard devices used in passthrough
VM1 (sophos utm), intel i211 NIC for wan
vm2 (truenas) onboard sata

Onboard device assigned to vmbr0 bridge - rtl8125. Also used for proxmox management.

Device under test
*intel ax200 wifi nic

During initial setup the intel wifi nic was enabled and assigned to a win10 and openwrt vm (bot not on at the same time). Other onboard devices passed through as indicated. Once I was satisfied I couldn't use the ax200 as an ap, it was disabled in the bios.

Trouble begain after rebooting. I no longer had video after proxmox started booting and the nic mapping got messed up. The i211 nic got reassigned as the proxmox management nic while the rtl8125 became unassigned. Some other amd device became assigned instead of sata passthrough.

I'm relatively new to proxmox but have several years of experience with esxi. My expectation was the existing config mapping (for already assigned present devices) would remain even after a device was disabled/removed.

Question: How can I lock in these mappings so enabling/disabling something doesn't blow up the whole server in terms of management.

 

[leesteken]

Proxmox uses the PCI-ID of devices, which on most motherboards shift up or down when adding or removing PCI(e) devices (physically or via BIOS settings). As far as I know, there is no automatic solution for this at this time. After a change to the number of devices, I enter the GRUB or systemd boot menu (pressing 'e') and change amd_iommu=on to amd_iommu=off to prevent passthrough (causing VMs to fail to start) and fix the numbers on the Proxmox console. If I 'm lucky to I remember beforehand, I disable autostart for each VM, because I also use those numbers in pre-start snippets.

 

[das1996]

Understood. Lesson learned. I've create a small script called maint_on.sh and maint_off.sh. This updates all vm's to autostart on and off respectively.

Regarding the amd_iommu=on parameter. I don't have that in /etc/default/grub nor the entries in /etc/modules (and specified in https://pve.proxmox.com/wiki/Pci_passthrough), __YET__ passthrough works¿? How? <confused>

 

[leesteken]

Understood. Lesson learned. I've create a small script called maint_on.sh and maint_off.sh. This updates all vm's to autostart on and off respectively.

Regarding the amd_iommu=on parameter. I don't have that in /etc/default/grub nor the entries in /etc/modules (and specified in https://pve.proxmox.com/wiki/Pci_passthrough), __YET__ passthrough works¿? How? <confused>

It is probably on by default nowadays (for Proxmox) but adding amd_iommu=off (when using AMD CPU) will still prevent passthrough from working.

 

[das1996]

Sure looks that way. I know I never touched grub or modules files (yet).

# lsmod | grep vfio
vfio_pci               53248  3
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       32768  2
vfio                   32768  10 vfio_iommu_type1,vfio_pci
irqbypass              16384  88 vfio_pci,kvm

 

[das1996]

@leesteken Re post #2. I run my pve headless. Surprisingly it's been running quite well. If I do need to do something on the server, I will install a basic video. I guess I need to remember to edit that iommu option first, before making any changes.