Link Aggregation - Two Access Ports on Switch Aggregated - Proxmox Aggs not working

[JakeBikeIT]

Hi -
I have two access ports on a switch that are aggregated on on a stacked pair for redundancy.
The switch manual claims this is 802.3ad LACP aggregation. However Proxmox doesn't seem to like this - Balance R-R was working but eventually some kind of loop occurred and the VMBR went off line for a bit. checking the logs shows this was some kind of loop

What other aggregation types might work and could this be an issue with the Nics themselves - as another device was working perfectly with Balance R-R same switch firmware and models and still seems to be working no problem.

Any help much appreciated.

 

[waltar]

Bond mode 0 balance-rr is not LACP which is mode 4. LACP must be enabled on networkcard ports and switch, all other modes are only on the cards.

 

[JakeBikeIT]

LACP mode is enabled on the switch ports - but the Proxmox connection does not seem to work with LACP

 

[waltar]

Balance-rr mode is wrong selected in proxmode.

 

[RichardH]

LACP mode is enabled on the switch ports - but the Proxmox connection does not seem to work with LACP

You need to create a bond on the host which is then used as a bridge-port for other virtual ports.

Something like:
auto bond0
iface bond0 inet manual
bond-slaves eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4

 

[JakeBikeIT]

That's exactly how it was configured but it quickly failed - I turned on Balance RR which I knew was incorrect - but it worked better - then stopped working - I am guessing XOR or TLB might give me better switch compatibility?

 

[RichardH]

any chance you can post both the switch config and the interfaces file from the host side?

 

[JakeBikeIT]

I've confirmed with the switch manufacturer that the mode on these ACCESS ports is indeed LACP - when I used this mode with any hash in proxmox I loose control over management and or the VMBR

 

[JakeBikeIT]

I could but I've tested all the LACP options in proxmox against the switch

 

[waltar]

Something like:
auto bond0
iface bond0 inet manual
bond-slaves eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4

I mean lacp runs on layer2+3 so I would outcomment and restart network with (take internal defaults without defining any wrong) :
Something like:
auto bond0
iface bond0 inet manual
bond-slaves eno3 eno4
# bond-miimon 100
bond-mode 802.3ad
# bond-xmit-hash-policy layer3+4

 

[JakeBikeIT]

Yeah that's a good idea.... however balance-tlb is working perfectly bandwidth and failover - we will monitor

 

[waltar]

For "perfectly working" you should test with real datatransfers (not iperf) best with 2 clients to server if you get more bandwitdh as with 1 port !!
I've seen so many bonded server networks in the past which didn't do more bandwitdh as 1 single connection ... just conserving the catX cables from doing their normal rates or whatever, looks nice from outside of the servers but without performance effect. It's easier to switch to higher technology in local parallel env (1g to 10g, or 10g to 100g opa/ib etc) without headache and 10x speedup (instead of 2x while bond 2x).

 

[JakeBikeIT]

For "perfectly working" you should test with real datatransfers (not iperf) best with 2 clients to server if you get more bandwitdh as with 1 port !!
I've seen so many bonded server networks in the past which didn't do more bandwitdh as 1 single connection ... just conserving the catX cables from doing their normal rates or whatever, looks nice from outside of the servers but without performance effect. It's easier to switch to higher technology in local parallel env (1g to 10g, or 10g to 100g opa/ib etc) without headache and 10x speedup (instead of 2x while bond 2x).

We are testing transfers that exceed the theoretical max of 1Gbe - however the redundancy is primary concern as its tolerates a switch failure in stack

 

[JakeBikeIT]

For "perfectly working" you should test with real datatransfers (not iperf) best with 2 clients to server if you get more bandwitdh as with 1 port !!
I've seen so many bonded server networks in the past which didn't do more bandwitdh as 1 single connection ... just conserving the catX cables from doing their normal rates or whatever, looks nice from outside of the servers but without performance effect. It's easier to switch to higher technology in local parallel env (1g to 10g, or 10g to 100g opa/ib etc) without headache and 10x speedup (instead of 2x while bond 2x).

Thanks for your input!! By the way