Enabling IOMMU Breaks Networking

[expnet]

When I add intel_iommu=on to my boot options in GRUB, all my networking (both for the host and the guests) stops working. I have to walk over to the machine, log in, and change it back.

At first glance, everything looks fine with the network (correct interfaces are showing as UP, bridges are still showing correct IPs, etc.), but nothing seems to be able to go in or out of the machine over the network.

Any ideas about what might be wrong (or what might be useful in debugging?)

 

[wolfgang]

Hi,

I guess you nic names have changed.
Please check your settings.

 

[expnet]

NIC names appear to be the same (enp6s0 and enp15s0)

 

[expnet]

So I took a closer look at this, and I did a diff of an "ip a" between iommu on and iommu off, and the only changes are the MACs of the tap/fwbr interfaces, which change every boot anyway.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:cd:2b:1a:80 brd ff:ff:ff:ff:ff:ff
3: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:cd:2b:1a:81 brd ff:ff:ff:ff:ff:ff
4: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UP group default qlen 1000
    link/ether 1c:1b:0d:f6:5c:0d brd ff:ff:ff:ff:ff:ff
5: enp10s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:cd:2b:19:f4 brd ff:ff:ff:ff:ff:ff
6: enp11s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:cd:2b:19:f5 brd ff:ff:ff:ff:ff:ff
7: enp14s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:cd:2b:19:6c brd ff:ff:ff:ff:ff:ff
8: enp15s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 00:0a:cd:2b:19:6d brd ff:ff:ff:ff:ff:ff
9: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1c:1b:0d:f6:5c:0d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1e1b:dff:fef6:5c0d/64 scope link
       valid_lft forever preferred_lft forever
10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0a:cd:2b:19:6d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::20a:cdff:fe2b:196d/64 scope link
       valid_lft forever preferred_lft forever
11: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr101i0 state UNKNOWN group default qlen 1000
    link/ether 4a:c6:55:a8:cc:0e brd ff:ff:ff:ff:ff:ff
12: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:3e:0e:47:2e:97 brd ff:ff:ff:ff:ff:ff
13: fwpr101p0@fwln101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 2e:80:12:1f:ad:7c brd ff:ff:ff:ff:ff:ff
14: fwln101i0@fwpr101p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i0 state UP group default qlen 1000
    link/ether 46:3e:0e:47:2e:97 brd ff:ff:ff:ff:ff:ff
15: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr101i1 state UNKNOWN group default qlen 1000
    link/ether 66:cd:77:a9:c5:69 brd ff:ff:ff:ff:ff:ff
16: fwbr101i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:28:17:22:a9:d6 brd ff:ff:ff:ff:ff:ff
17: fwpr101p1@fwln101i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 92:d5:6c:59:a7:c7 brd ff:ff:ff:ff:ff:ff
18: fwln101i1@fwpr101p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr101i1 state UP group default qlen 1000
    link/ether 02:28:17:22:a9:d6 brd ff:ff:ff:ff:ff:ff

 

[expnet]

Anyone have other ideas on this?

 

[dcsapak]

can you post the 'dmesg' output of a boot with and without iommu ?

 

[nigg]

I have a similar problem so instead of opening a new issue i first try to add here.
System is Ryzen 3200G (APU) I try to pass a PCIe NIC to a VM (kvm). When I start that machine my network stops working. The onboard NIC (Realtek) where BR0 is running on works fine until the start of that VM with the Intel PCIe NIC attached. IOMMU on MB is activated and I loaded Kernel modules. Right now I have no idea where to look for that Issue. I don't see the connection between the network BR0 and the VM with attached PCIe device. Is there an obvious reason or does anyone have a pointer where to investigate??? I am really at a loss here. I actually built that system for a cost efficient firewall + server solution.
dmesg gives no additional information on vm start with subsequent network failure.

 

[flakmoppen]

Same. I was following a youtube tutorial on how to virtualize your router but ran into problems.
I'm running a freshly installed Proxmox on a HPE Microserver Gen10+. No VMs other than an uninstalled pfsense. When I add "intel_iommu=on" to my boot options in order to pass along the NICs, and then try to boot up to install pfsense, it immediately looses networking.
And other than my router not seeing that it is still connected, the machine is running just fine... but without networking.
If I remove the NIC's from the VMs hardware list OR remove "intel_iommu=on", networking is back.

Looking around for a solution I saw a few people having a similar issue when different devices shared the same iommu group, but according to find /sys/kerneliommu_groups/ -type l that's not the issue in my case.
Output:

[...]
/sys/kernel/iommu_groups/10/devices/0000:00:1d.2
/sys/kernel/iommu_groups/11/devices/0000:00:1d.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.5
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.1
/sys/kernel/iommu_groups/15/devices/0000:02:00.2
/sys/kernel/iommu_groups/16/devices/0000:02:00.3
/sys/kernel/iommu_groups/17/devices/0000:01:00.0
/sys/kernel/iommu_groups/18/devices/0000:01:00.1
/sys/kernel/iommu_groups/19/devices/0000:01:00.2
/sys/kernel/iommu_groups/20/devices/0000:01:00.4

(13-16 are the NICs)
I still tried adding "pcie_acs_override=downstream,multifunction" just in case, but it made no difference.

 

[flakmoppen]

Found two ways of starting up the pfsense install without Gen10+ going offline:
1. Creating more network bridges for the other NICs and using them instead, or
2. Adding them as PCI Devises without All Functions checked.
I'm not sure what All Functions does in this case exactly, but it seems to me that everything is working just fine without it.

 

[Azabua]

Same problem here, after i add intel_iommu=on to GRUB, all my networking stops working, not even showing up in my router any more
The NIC seams to have the same name as before

 

[saintgeele]

I am looking for more support on this issue. Can anyone help?

 

[leesteken]

I am looking for more support on this issue. Can anyone help?

If it's not because of shared IOMMU groups, then maybe iommu=pt might help as it uses the same mapping as without IOMMU for non-passed through devices. For some Intel integrated graphics, it's possible to exclude that device from IOMMU, maybe something like that is possible for other devices?

 

[saintgeele]

If it's not because of shared IOMMU groups, then maybe iommu=pt might help as it uses the same mapping as without IOMMU for non-passed through devices. For some Intel integrated graphics, it's possible to exclude that device from IOMMU, maybe something like that is possible for other devices?

I have iommu=pt enabled as well.