second NIC passthrough broke after adding bridge

[Rayures]

Hi there,

i have a clean proxmox 6.2 install (efi), activated iommu, created a VM (pfsense in this case) and selected 2 NICs for passthrough (seperate iommu groups).

01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

I selected 2 NIC for passthrough and installed pfsense. This works fine. Both nics can be used in the VM. so far so good.

Nov 05 13:19:02 proxmox systemd[1]: Started 100.scope.
Nov 05 13:19:02 proxmox kernel: vfio-pci 0000:06:00.0: enabling device (0002 -> 0003)
Nov 05 13:19:02 proxmox kernel: vfio-pci 0000:05:00.0: enabling device (0002 -> 0003)
Nov 05 13:19:03 proxmox pvedaemon[1413]: <root@pam> end task UPID:proxmox:00000888:000033FF:5FA3EDB5:qmstart:100:root@pam: OK

But then i added a bridged device (virtio) to the VM and things break.

Nov 05 09:13:14 pve kernel: vfio-pci 0000:06:00.0: enabling device (0002 -> 0003)
Nov 05 09:13:14 pve kernel: vfio-pci 0000:05:00.0: enabling device (0002 -> 0003)
ma_map(0x7fc90a267580, 0xc1645000, 0x3000, 0x7fc90a61c000) = -14 (Bad address)
Nov 05 09:48:43 pve QEMU[2144]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 09:48:43 pve QEMU[2144]: kvm: vfio_dma_map(0x7fc90a267580, 0xc1600000, 0x20000, 0x7fc908a3d000) = -14 (Bad address)
Nov 05 09:48:43 pve QEMU[2144]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 09:48:43 pve QEMU[2144]: kvm: vfio_dma_map(0x7fc90a267580, 0xc1641000, 0x3000, 0x7fc90a618000) = -14 (Bad address)
Nov 05 09:48:43 pve QEMU[2144]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 09:48:43 pve QEMU[2144]: kvm: vfio_dma_map(0x7fc90a267580, 0xc1600000, 0x20000, 0x7fc908a3d000) = -14 (Bad address)

When this happens i cannot undo it.
-reboot - error persists
-removing the bridge device - error persists.
-creating new vm (tried pfsense and ubuntu) and passthrough 2 nics - error persists
-checked different proxmox config files, could not find anything.

Only thing that works is a clean proxmox install. Then passthrough works again.

after a clean install the pve packages are:

proxmox-ve: 6.2-1 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Then after a full update tot 6.2-15 it breaks.

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-4
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-18
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Nov 05 14:55:24 pve1 kernel: vfio-pci 0000:06:00.0: enabling device (0002 -> 0003)
Nov 05 14:55:24 pve1 kernel: vfio-pci 0000:05:00.0: enabling device (0002 -> 0003)
Nov 05 14:55:24 pve1 pvedaemon[1445]: <root@pam> end task UPID:pve1:00000694:00000D11:5FA4044B:qmstart:200:root@pam: OK
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: vfio_dma_map(0x7f076c667580, 0xc2000000, 0x20000, 0x7f076ae5d000) = -14 (Bad address)
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: vfio_dma_map(0x7f076c667580, 0xc2021000, 0x3000, 0x7f076ca14000) = -14 (Bad address)
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: vfio_dma_map(0x7f076c667580, 0xc1e00000, 0x20000, 0x7f076ae3d000) = -14 (Bad address)
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: VFIO_MAP_DMA failed: Bad address
Nov 05 14:55:25 pve1 QEMU[1706]: kvm: vfio_dma_map(0x7f076c667580, 0xc1e21000, 0x3000, 0x7f076ca10000) = -14 (Bad address)

something is wrong with an update....