[SOLVED] VM multicast VRRP packets drop

[Samuele Bistoletti]

Hi,
I'm experiencing a strange problem, not sure if it is some way related to Proxmox, but it started to happen this night after 5.4-13 upgrade on our nodes.

As I said, this night we have updated our nodes to the latest Proxmox 5.x version (5.4-13). After nodes update some monitoring systems started alerting us for inbound packets drop on network interfaces, on all VMs in all VLANs. After some investigation I discovered that there was exactly 1 drop per second, so I decided to start tcpdump in order to capture some packets on one Ubuntu 18.04 cloud init VM (strange thing was that while tcpdump was capturing packets no drops were reported to the corresponding network interface). I analyzed the output with Wireshark and discovered that dropped packets were of VRRP type coming from our main router, and this is correct because Pfsense send multicast VRRP packets over the network for its HA (Carp). What I'm not able to explain to me is why this problem started to happen after Proxmox update, and not before. No other updates were applied this night, no updates on VMs and no updates over the network, only Proxmox itself.
This drops are not affecting our systems, but I would like to understand the nature of this anomaly and possibly stop dropping packets.

Anyone has experienced this or similar issue?

I also noticed that from firewall tab on VM Proxmox GUI is possible to decide to allow or block router advertisement, can I block VRRP packets this way on hypervisor level?
During packet capture I also saw spanning tree advertisement packets coming from our switches, in a cloud environment I think it would be a good practice to block this packets before thery reach the VM. Is that right?

Thanks in advance for your help.

 

[Samuele Bistoletti]

Update:
I tried to block VRRP packets with the firewall from proxmox gui. I have the same packet drop, also without VRRP packets incoming on VM net. So it's not pfsense fault. I ran another tcpdump but there is not any relevant packet, traffic is normal (ssh + postgresql client).
Any ideas on how to diagnose this problem?

Thanks

 

[Samuele Bistoletti]

Update 2:
I tried another packet capture inside the VM. I reduced the traffic at minimum with firewall rules, and the only thing I can't block is spanning tree bpdu packets incoming on net interface. Can it be the cause of packet drop?
I attach a screenshot from wireshark view of one of this packets.

 

[Samuele Bistoletti]

Problem solved! I disabled STP on switch side, only on Proxmox LAN ports. Drop is now 0. The strange thing is that I have 2 physical machines on the same VLAN, with the same OS (Ubuntu 18.04), with all updates installed, and drop was 0 also before I disabled STP. No idea why.