Bonding LAG with multiple switches

[klowet]

Hi

I have a question about bonding and HA. I want to create a HA PVE cluster, but I am confused with the bonding and its mode. See this simplified picture. I have two switches (Mikrotik CRS317, not stackable) and multiple PVE nodes (just one signed).

What should I configure to create a HA network? The switches do support LACP, Active-Passive, etc. LAG protocols. But they don't support stacking nor multi-chassis LAG), so they are two separated L2 devices.

In PVE, I can choose the LACP mode, but am I right that I can not use this because I can not create a LAG group on the two different switches, because I have to configure LACP also on the switches?
If I use Active-Passive eg, can I just configure that and don 't configure something on the switches?

Thanks

 

[wolfgang]

Hi,

You can use LACP over two switches if they support MC- LAG or equal.
see
https://en.wikipedia.org/wiki/MC-LAG

If I use Active-Passive eg, can I just configure that and don 't configure something on the switches?

Yes this works if you have the switches connected as you show in your pic

 

[klowet]

Hi

The switches don't support MC-LAG, they don't stack.
So my best option is to use Active-Passive bonding on PVE and "normal links" without any sort of bonding on both switches. Is that right?

 

[wolfgang]

s that right?

Yes

 

[klowet]

Okay, thanks.
What with the other modes, like balance-alb? Does that has to be configured on the switches, or is it possible in this scenario to use that mode?

 

[wolfgang]

Does that has to be configured on the switches, or is it possible in this scenario to use that mode?

It does not work in all cases so I would not use this.

 

[klowet]

Clear, thx!

 

[Dalamar]

Hi, I know this is an old post, but i have a similar situation here; one proxmox Host, with 2 non-stackable L2 switches, with support for LACP each of them. Proxmox host has 4 GB ports and i would like to have both HA and at the same time, the max amount of bandwidth possible for the VM guests.

I am looking for an option to be able to use up to 2Gpbs for all the vms while having HA, fully loosing one switch will divert all the traffic through the other switch.

The only idea i have come up with is

Proxmox Bridge0 connected to bond0, LACP to Switch0, 2 Gbps ports
Proxmox Bridge1 connected to bond1, LACP to Switch1, 2Gbps ports

VM Linux guests connected to both bridges, using Linux bonding active/passive

However i am afraid if the switch goes down, the related bridge will be anyway up and the Linux guests will not be aware of the problem. Any ideas or suggestions to get this done?

Obviously i can always configure VMs manually to be connected to a different bridge each, but i was hoping to find out a less manual way

Best
D.

 

[wolfgang]

Try a nested Bond.
Here is a sample config for it.
But do testing before you use this in production.

auto lo
iface lo inet loopback

iface ens18 inet manual

iface ens19 inet manual

iface ens20 inet manual

iface ens21 inet manual

auto bond0
iface bond0 inet manual
      slaves ens18 ens19
      bond_mode 802.3ad
      bond_miimon 100
      bond-xmit-hash-policy layer2+3

auto bond1
iface bond1 inet manual
      slaves ens20 ens21
      bond_mode 802.3ad
      bond_miimon 100
      bond-xmit-hash-policy layer2+3

auto bond3
iface bondha inet manual
      slaves bond0 bond1
      bond_mode active-backup
      bond_miimon 200

auto vmbr0
iface vmbr0 inet static
      address  192.168.1.2
      ntmask  255.255.255.0
      gateway  192.168.1.1
      bridge-ports bond3
      bridge-stp off
      bridge-fd 0
      bridge-vlan-aware yes
      bridge-vids 2-4094

 

[Dalamar]

Sorry for the delay on answering, i have been doing some tests with this configuration, and i can confirm it works flawlessly.
I can mention some warnings based on the problems i have experienced during the setup

-After rebooting the proxmox node -for other reasons- the /etc/network/interfaces file was rewritten with the default configuration. Maybe i had pending changes to apply from the WEB UI, but i had to apply all changes again. Now i have a copy of the file, i have also removed write permissions and after a second reboot it was not rewritten.


-Doing the changes with containers or VMs on, did not work well for me in the second server (i have done it in two proxmox nodes) . While network configuration in LXCs or KVMs was not changed, just changing the configuration and restarting network service resulted in network outage for all VMs and Containers, while the main server ip was reachable, and responding to ping and ssh. A server reboot fixed the issue.

-Once modified, nework config can not be managed by WEBUI, you will obtain errors related to interface names, it seems to me the issue is web UI does not support nested bonding.

In my tests simply shutting down all the ports, shutting down one of the bond interfaces or physically disconnecting cables from the switch was properly detected and traffic was diverted to the available bond interface, which is great.

In case of one single cable failure, traffic was diverted to the other ports inside the same bond interface, which is expected as the lacp group is still on.

Thanks Wolfgang for the help!
The nodes are still in testing for the next week or two before being put into production.