[SOLVED] VMs are losing network access for a few seconds. Host does not.

[xdanik]

I am having issue with one of my Proxmox nodes. VMs are losing network access for a few seconds (1 - 5). Host does not.
By "losing network access" I mean they cannot reach (or be reached) other devices on the same subnet.
Each VM drops connection at different time.

Host is Dell R320 running up-to-date Proxmox 7.2-11 and its connected using single build-in NIC (NetXtreme BCM5720) to Unifi USW-24-G1 switch.
This host runs 3 virtual machines - TrueNAS core and two instances of Debian 11.
All VMs have static IP on VLAN 25 - subnet 192.168.25.0/24
Host itself is connected VLAN 16 - subnet 192.168.16.0/24

Hosts /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.16
iface vmbr0.16 inet static
        address 192.168.16.1/24
        gateway 192.168.16.254

(One of) Debian VM config:


This is output from Mikrotik router (running on the same subnet) that I use to monitor the connectivity:


I have verified that all the VMs have unique mac addresses on the network.
I also verified the network for any IP conflicts (running arpmon).

There seems to be nothing in the host's syslog or in the VMs logs. Also nothing in the Unifi switch logs.

Any ideas for what to look for please?

 

[spirit]

can you check the size of mac-address-table on your unify switch ?

if you use "bridge-vids 2-4094" and than you unift switch allow all vms, it's possible that you'll 1mac for each vlan.

(I have read than unify switch are limited to around 16000 mac entries)

 

[xdanik]

Yes, I was able to list mac-address-table on the unifi switch when one of the VM was unreachable. And indeed the mac address of the VM was missing!
Few second later and the mac address was back...

What could cause VM mac address to disappear on the switch?
AFAIK the mac should be only dropped in case of timeout which should be 300 seconds.

I have already tried connecting the host to different port on the switch and restarting the switch.

This is home environment - the switch "knows" less than 100 mac addresses, so 16k limit should not be an issue.
There are other devices connected to the switch (including another unifi switch) and they work with no issues.

 

[xdanik]

The switch has (R)STP disabled and there are no loops in the network.

 

[spirit]

What could cause VM mac address to disappear on the switch?
AFAIK the mac should be only dropped in case of timeout which should be 300 seconds.

should be related to timeout.
I suggest to try to increase it to 30min or 2h.

5min can be too short, as arp cache timeout on a vm is generally around 5min.

 

[xdanik]

I have just noticed that the mac address sometimes disappears from the mac address table and sometimes just changes to completely different port.
It changes from port 02 where the host is connected to port 21 where the Mikrotik router is connected.
I have no idea why it would do this.
The Mikrotik bridge mac address table shows no changes.

I have disabled HW offload to the bridge port on Mikrotik and it seems the problem went away.
The Mikrotik is RB5009UG+S+ running RouterOS 7.6.

 

[linum]

Since I noticed the same problem (VM loosing connection, host still works) I wonder if your problem is really solved or not...

 

[xdanik]

Yes, all issues seems to be to gone after disabling the HW offload on the Mikrotik router.