[SOLVED] Firewall failover shuts down the fibre box

[j.hedin]

Hi

I have a very strange problem. I have a 3 node Proxmox cluster at home. The network is a ring of 10GB. Two of the nodes are connected to local LAN. All this handled by RSTP and it works well (openvswitch and Unifi play along nicely). For the internet connection, all three nodes are connected to the fibre box via an unmanaged TP-link switch. Internally in the Proxmox there is an additional openvswitch bridge that filters away any RSTP. To complicate things a bit further, there is a Debian trafic shaper in front of the firewall since the OpenBSD QoS is not as good as the Linux one.

 3 x Proxmox
+--------------------------------------------------+
| +---------+              +--------+              |               +---------+   +-----------+
| | OpenBSD |--> vmbr0 --> | Debian | --> vmbr1 ---+--- Ethernet --+ TP-link +---+ Fibre Box |
| +---------+              +--------+              |               +---------+   +-----------+
+--------------------------------------------------+

This means that the Debian machine will answer on the OpenBSD MAC on vmbr1.

Now for the problem. If I do a live migration, or even if a do a stop/migrate/start of the Debian load balancer, the WAN led on the fibre box sometimes goes blank. The only way to recover is to restart the fibre box. How can the fibre box even know that the MAC of the FW has moved between two ports on the dumb switch?

 

[j.hedin]

For a reference about the openvswitch and RSTP. It works really stable now.

https://forum.proxmox.com/threads/open-vswitch-ovs-with-rstp-and-ovsintport-arp-issue.140120/

 

[j.hedin]

Update: I can migrate the FW on OpenBSD without any problems. The VLAN used on vmbr0 is trunked via the 10GB ring.

 

[j.hedin]

Update: I think i found the reason. On live migration, someone sends a rarp with the MAC of the Debian QoS bridge interface.

22:07:13.026437 ARP, Reverse Request who-is d2:fb:6f:af:67:88 tell d2:fb:6f:af:67:88, length 46

This makes the fibre box think that there are 2 MAC on the fiberbox and blocks the port. Just need to find out how to prevent live migration to send rarp for this machine.

 

[j.hedin]

Update: Offline migration does not send RARP that confuses the fiberbox.

 

[j.hedin]

Found a solution

ovs-ofctl add-flow vmbr1 rarp,actions=drop