VM connectivity lost after bond failover

A

apollo13

Renowned Member
Proxmox Subscriber
Jan 16, 2018
67
9
73
Hi there,

I am seeing a loss of connectivity after bond failover of an active-backup bond to two switches. My setup is as follows:

Code: 
+-------+                                +-------+
|switch1|----(inter switch lacp trunk)---|switch2|----client
+-------+                                +-------+
    |                                        |
    | primary      +-------+          backup |
    ---------------|proxmox|------------------
                   +-------+

The LACP inter switch connection is working fine and connectivity with only a single bond slave as well.

My bond configuration is basically this:
Code: 
auto enp65s0f0
iface enp65s0f0 inet manual

auto enp65s0f1
iface enp65s0f1 inet manual

auto bond2
iface bond2 inet manual
    bond-slaves enp65s0f0 enp65s0f1
    bond-miimon 100
    bond-mode active-backup
    bond-primary enp65s0f0
#VM-01

auto vmbr0
iface vmbr0 inet manual
    bridge-ports bond2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#VM-BRIDGE-01

Now the following seems to happen / is used for testing:
* The client pings a VM on the proxmox server
* I pull the primary cable to test the failover
* /proc/net/bonding/bond2 confirms the bond fails over properly
* Ping no longer gets answers
* Looking into the mac address table on switch2 I see that the MAC of the VM is still bound to the inter switch trunk
* Reattaching the primary cable results in ping to get answers again

This kinda makes sense since the MAC address table on the switch has not had a reason for an update yet. So what to do? One thing I can imagine is utilizing qemu's monitor feature to send an announce_self for all the VMs running when the bond fails over (I think qemu does this after live migration as well). This would result in GARP brodcasted and would force all switches in the broadcast domain to update their records.

* Are those assumptions correct?
* Assuming they are correct are there any good ways to monitor bond failover (short of polling /proc/net/bonding/bond2) so I can script this myself?
* Is this something which would make sense to add to proxmox itself? The setup on it's own doesn't seem to be that uncommon; so I am wondering why noone else is seeing this (or are they and I just couldn't find it in the forum).

Thanks,
Florian
 
I have managed to work around this problem using the following simple python script:
Python: 
import socket
import pathlib
import time
import sys
import pyroute2
import logging

QEMU_SERVER_PATH = "/var/run/qemu-server/"


def announce_self(qmp_path):
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock:
        sock.settimeout(0.1)
        sock.connect(str(qmp_path))
        sock.recv(1024)  # read initial message
        sock.sendall(
            b'{"execute":"qmp_capabilities"}\n{"execute":"announce-self","arguments":{"initial":50,"max":550,"rounds":4,"step":50}}'
        )
        sock.recv(1024)  # read first command
        sock.recv(1024)  # read second command


def announce_all():
    for p in pathlib.Path(QEMU_SERVER_PATH).iterdir():
        if p.name.endswith(".qmp"):
            try:
                announce_self(p)
            except Exception as e:
                logging.error(f"Failure announcing {p}: {e}")


if __name__ == "__main__":
    logging.basicConfig(
        format="%(asctime)s [%(levelname)s] %(message)s", level=logging.INFO
    )
    if len(sys.argv) < 2:
        print("Usage: bond_watch.py BOND", file=sys.stderr)
        sys.exit(1)
    ifname = sys.argv[1]
    active_slave = None
    ipr = pyroute2.IPRoute()
    logging.info(f"Starting bond watch on {ifname}")
    while True:
        link = ipr.get_links(ifname=ifname)[0]
        ifinfo = link.get_attr("IFLA_LINKINFO")
        bond = ifinfo.get_attrs("IFLA_INFO_DATA")[0]
        new_slave = bond.get_attr("IFLA_BOND_ACTIVE_SLAVE")
        if new_slave != active_slave:
            if active_slave is not None:
                logging.info("Detected failover")
                announce_all()
        active_slave = new_slave
        time.sleep(1)

It is rather inefficient and I am wondering if I somehow can get events from the kernel.
 
Hi,

I have this exact issue, did you find any solution for it?

It is kind of a show stopper for us to start using proxmox compared to our vmware setup to day.

Thanks
 
LACP is not an option for us. we are using Cisco UCS fabric interconnects.
And using selfmade scripts for production setups is not a good solution.

Did you ever find any answers on why proxmox is designed this way?
 
LACP is not an option for us. we are using Cisco UCS fabric interconnects.
Click to expand...
That is mistake number one :D

And using selfmade scripts for production setups is not a good solution.
Click to expand...
Others would say that the upside of an open system like proxmox is that you can do stuff on your own. And I'd trust my selfmade scripts more than VMWare ;)

Did you ever find any answers on why proxmox is designed this way?
Click to expand...
It is not "designed" in this way. It is probably simply the default behavior you'd expect from the OS. For everything else proxmox would have to add code. So it is probably just that noone ran into this "problem".
 
Using UCS fabrics? or option out LACP?
I would like to control the traffic flow, I'm not 100% in control if I use LACP
I will not like to have traffic flowing through the fabrics uplinks unless it is a failover scenario.
at least not in HCI setups.
On VM traffic, sure I like that outside my fabric, but no mather what mode I'm using, the mac is not moving in failover scenarios.

Sure, you are right, and I will test your script. - I'm just used to enterprise level, but now that vmware don't want us as customer anymore, I need to open my eyes....

I'm surprised that none have this problem, it is a common way of designing network around HCI fx. Cisco hyperflex on UCS fabrics.
At some point I would expect proxmox to make this, since they call it enterprise.....

VMware have this behavior for linux vm's that have sub interfaces within (so really a OS behavior on the vm). but proxmox have this behavior on hypervisor level, in that case, I would expect proxmox to make the fixes.
 
Generally this is a known limitation of active-backup together with bridges, people run into it from time to time. The general solution, as already mentioned, is LACP. In your specific case, after skimming the UCS documentation (I have 0 experience with it!), apparently balance-tlb and balance-alb also seem supported. I'm not sure if they suffer from the same drawbacks, since everything seems to be managed from the UCS side, but it might be worth looking into it?

For the active-backup failover GARP, you can open a feature request over on our Bugzilla! [1]

[1] https://bugzilla.proxmox.com
 
Thanks for your answers.

I have swapped to open vSwitch, and the failover just works as it should with active/backup.
also must faster failovers.

I'm not done testing, but it looks to be the solution with UCS fabrics.

regards balance-tlb and balance-alb, they have the same limitations. - also outside the bridges, the failover works but it is slow

LACP nor vPC is not an option for UCS fabrics, since the two swtiches not have any network between them beside via the core network.
And the thing with LACP, it is fine for VM traffic. but when you go with HCI like ceph, hyperflex etc. you don't use LACP or vPC between two switches, because you can have storage traffic flowing through the core, and that adds latency, so isolate it to switch a or b and use the core in failover scenarios.

To add complexity, we are working on a "best effort" cluster with some HP servers via appliance ports on UCS fabrics and that just add headaches regards fast failovers.
When it comes to Cisco servers on UCS fabrics, the fabric "helps" the mac table to be moved a bit faster to the other switch.

I can update you guys later when I have migrated the whole cluster to open vSwitch and when I have done some testing?
And fill free to comment on you experience with open vSwitch. - I would love to get some knowledge
 
You must log in or register to reply here.
Top Bottom
Back