[SOLVED] pve LACP with HP Switch

[Andrew31]

Participants: two PVEs (6.1-8 ) and an old HP-2810-24G (J9021A)

I want to have both failover and double bandwidth on this two fresh installed PVEs .
But actually works only failover, the bandwidth is limited to 1Gbps.
I am sure I am missing something, this is why I am posting here.


pve1

Spoiler: /ETC/NETWORK/INTERFACES

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr1
iface vmbr1 inet static
address 192.168.16.44/24
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

pve2

Spoiler: /ETC/NETWORK/INTERFACES

auto lo
iface lo inet loopback

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto bond0
iface bond0 inet manual
bond-slaves enp5s0f0 enp5s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 192.168.16.102/24
gateway 192.168.16.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

HP-2810-24G

Spoiler: SHOW RUNNIG-CONFIG

node0-SW# sh running-config

Running configuration:

; J9021A Configuration Editor; Created on release #N.11.78

hostname "node0-SW"
time timezone 2
interface 5
name "LAN1-pve2"
lacp Active
exit
interface 6
name "LAN2-pve2"
lacp Active
exit
interface 10
disable
exit
interface 11
name "LAN3-PVE1"
lacp Active
exit
interface 13
name "LAN4-PVE1"
lacp Active
exit
interface 23
disable
exit


snmp-server community "public" Unrestricted
vlan 1
name "DEFAULT_VLAN"
untagged 1-24
ip address dhcp-bootp
exit
vlan 16
name "16net"
tagged 2
exit
vlan 22
name "22guest"
tagged 2
exit
vlan 111
name "VLAN111"
tagged 1-3
exit
no stack
spanning-tree
spanning-tree priority 0
ip ssh

Spoiler: SHOW LACP

LACP

PORT LACP TRUNK PORT LACP LACP
NUMB ENABLED GROUP STATUS PARTNER STATUS
---- ------- ------- ------- ------- -------
5 Active Dyn1 Up Yes Success
6 Active Dyn1 Up Yes Success
11 Active Dyn2 Up Yes Success
13 Active Dyn2 Up Yes Success

 

[spirit]

lacp can(t use more than 1 link for 1 tcp connection. so you need multiple connections to test full bandwith.

also, with layer2+3, the tcp connections are balanced, from a hash : src-ipsrc-ipdst, so if you test between 2hosts, even with multiple connections, you'll still use 1 link.

you need layer3+4, to hash srcip-srcport-destip-desport. then it'll work with multiple links between 2 hosts.

also, the hash algorithm is for oubound traffic from the proxmox host. You need to do the same on your physical switch, for the traffic outbound from the switch. (so inbound traffic to proxmox host)

 

[Andrew31]

@spirit Thank you for guidance on solving this.
As you suggested LACP depending on hash policy L2, L2 + L3, L3 + L4 needs multiple connections.
I modified to L3 + L4 and tested with iperf (2 servers + 2 clients, same IP different TCP ports) - without success.

After that I found HP detailed manual of my switch and figured out that this generation is able to use only L2 hash policy (source mac + destination mac) and changed proxmox back to L2 + L3 and created two VMs on each PVE and repeated test, and finally was able to achieve 2 Gbps per Dyn link.

For those who also had similar experience with old HP switches, only these generations (3500/6200/5400/8200/6600) and newer have hash policy of L3 + L4.

For now I will use this config, but planning to upgrade switch when all this troubles with covid will calm down ))

 

[linux]

For now I will use this config, but planning to upgrade switch when all this troubles with covid will calm down ))

Having done some research into this today, the L3+4 configuration isn't fully RFC-compliant as data can arrive out-of-order.

By default, the HP switch configurations (where they can handle a hash policy at L4, but are not configured to) will use L3, with fallback to L2.

So the Proxmox-recommended configuration of L2+3 seems quite safe in that sense. Yet to properly test all of this though - so beware!