[SOLVED] No network after adding VirtIO NIC to a VM already having a PassThrough NIC.

[proxdox]

Hi,

Proxmox Version: 8.4.14

Dealing with a strange issue on a new install of an latest Alpine linux VM, here is what I did:

Passthrough enp2s0 ((02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)) OK.
eth0 static IP (192.168.1.5) survives reboots, CAN ping from another computer. OK.
$ poweroff
Add new NIC (vmbr0, virtIO) /// vm conf shows:: net0: virtio=AC:2A:12:87:A6:26,bridge=vmbr0,firewall=1 /// OK.
< Start >
Now, eth0 doesn't work. CANNOT ping eth0 IP (192.168.1.5) from another computer. FAIL
Gateway IP (192.168.1.1) is valid and others can ping it. So, try to ping the gateway from eth0, NOTHING. FAIL
$ poweroff
Remove net0 virtIO nic
< start >
eth0 works fine again!

>> Also, in another iteration, tried hot plug/add the VirtIO nic (paravirtualized or other), while VM running and configure IP to what is now eth1.
eth0, eth1 both work fine and ping/respond OK. Reboot, all fails again. eth0 and eth1 won't ping / respond. Shutdown, remove VirtIO nic, eth0 starts working again!

Tried multiple iterations/installs and narrowed the process to above.
Adding virtIO nic (just add, no config and doesn't matter if it it VirtIO, E1000, E1000E, RTL8139...) fails whole network on this VM only.
Other VMs that are on this bridge(vmbr0) work perfectly fine!
Also, I have 2 other nics (pci slot) via passthrough in a pfSense VM and they work fine.

What did I miss?

 

[Onslow]

Hi. The PVE itself has no network problems during these experiments, right?

Then show us the outputs of commands at the VM:

cat /etc/network/interfaces
ip a
ip neighbour
ip route

in "OK" and in "FAIL" situations. Inside "CODE" tags

And similar network details of the "another computer".
Is there DHCP? If yes, what is the IP addresses range?
Is there any VLAN in your network?

Edit: maybe cat /etc/network/interfaces from the PVE as well.

 

[proxdox]

First, I totally appreciate you helping me!
Second, I am not sure it is related to regular ip "network" config (i.e. ip, gateways, subnets, duplicate mac, routes etc.)

Here is what I found:








There was one very SIMILAR issue I found on the networking forum. I haven't gone through it all yet.
However, I did add the extra ".0" to make sure of full address. It now shows as : hostpci0: 0000:02:00.0 in the vm conf file.
Regardless, it made no difference in the behavior.

Also, after more observation, it does seem to take down PVE network as well. As I type this, the whole bridge, VMs all lost connectivity as I was tinkering with Alpine Linux vm. So, it strongly points to IOMMU / passthrough...

 

[proxdox]

Just to add:

dmesg | grep -e DMAR -e IOMMU

[    0.007135] ACPI: DMAR 0x000000007F4989F0 0000A8 (v01 INTEL  EDK2     00000002      01000013)
[    0.007155] ACPI: Reserving DMAR table memory at [mem 0x7f4989f0-0x7f498a97]
[    0.074014] DMAR: IOMMU enabled
[    0.200108] DMAR: Host address width 39
[    0.200110] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.200120] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.200123] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.200127] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.200130] DMAR: RMRR base: 0x0000007f804000 end: 0x0000007fa4dfff
[    0.200133] DMAR: RMRR base: 0x00000088800000 end: 0x0000008affffff
[    0.200135] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.200138] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.200140] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.203419] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.560650] DMAR: No ATSR found
[    0.560653] DMAR: No SATC found
[    0.560655] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.560656] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.560658] DMAR: IOMMU feature nwfs inconsistent
[    0.560660] DMAR: IOMMU feature pasid inconsistent
[    0.560663] DMAR: IOMMU feature eafs inconsistent
[    0.560665] DMAR: IOMMU feature prs inconsistent
[    0.560667] DMAR: IOMMU feature nest inconsistent
[    0.560669] DMAR: IOMMU feature mts inconsistent
[    0.560671] DMAR: IOMMU feature sc_support inconsistent
[    0.560673] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.560676] DMAR: dmar0: Using Queued invalidation
[    0.560682] DMAR: dmar1: Using Queued invalidation
[    0.561210] DMAR: Intel(R) Virtualization Technology for Directed I/O

Also changed the GRUB_CMDLINE from default to see if it would fix:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"

Still the same thing.

 

[proxdox]

Got it resolved finally and here is what I did:

1. Remove all network interfaces on the affected VM.
2. Power off PVE and restart. ( clear my mind )
3. Add VirtIO nic first to the VM > start > configure as eth0 ip addr.
4. Shutdown VM.
5. Now add PCI passthrough nic to VM > start > configure eth1 ip addr.
6. Test everything.
7. Shutdown and restart VM > All good!

Basically, what I understood is that the adapter assignment is jumping and causing this issue. This may happen due to the initialization of wrong adapter first? Not entirely sure but it seems to fit my current working theory and yes, it was as simple a fix as that!

Anyway, I hope this helps someone later.

 

[Onslow]

That's why I asked you to show

cat /etc/network/interfaces
ip a
ip neighbour
ip route

I was wondering why you didn't ;-).

 

[proxdox]

That's why I asked you to show

cat /etc/network/interfaces
ip a
ip neighbour
ip route

I was wondering why you didn't ;-).

Hey,
It's not that I didn't want to. Actually, I had a draft with all of it and ready to post but in that process, everything hung on VMs / PVE bridge. And, that's what triggered me to re-evaluate what's happening (see my screenshots) and got buried.
Prior to that, all the info you were asking for was showing up properly in the affected VM, and PVE. Funny thing is, it looks and works fine in the VM like I stated in the first post but, it definitely fails after VM reboots and the whole thing becomes erratic afterwards eventually taking down the bridge and losing network PVE wide.

Anyway, it is stable now and of course, thank you!