802.3ad, no partner PDU

[phastier]

Hi all,

I have a cluster with 6 members, linked to a Ubiquiti Edgeswitch. On each members, I have setup two links, and whilst round-tobin works just great; I wanted to go to 8023.ad. I give the configuration on the Proxmox side below. On the switch side, I do have 3 non-proxmox members doing 802.3ad with no issue.

All proxmox members do not show parnter lacp pdu, and on the switch the side, the link is not seen as active. I went through the forums a lot, but honestly, I'm lost now. It should be simple.

Proxmox node configuration :

auto ens9
iface ens9 inet manual
mtu 9000

auto enp3s0f0
iface enp3s0f0 inet manual
mtu 9000

auto bond0
iface bond0 inet manual
bond-slaves enp3s0f0 ens9
bond-miimon 100
bond-mode 802.3ad
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 192.168.0.73/24
gateway 192.168.0.2
bridge-ports bond0
bridge-stp off
bridge-fd 0
mtu 9000

State of bond :

root@pve3:/etc/network# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.2.9-1-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 8e:9e:1a:00:be:49
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 9
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp3s0f0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 98:5a:eb:dc:1c:45
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8e:9e:1a:00:be:49
port key: 9
port priority: 255
port number: 1
port state: 77
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

Slave Interface: ens9
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 78:7b:8a:cc:c2:83
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8e:9e:1a:00:be:49
port key: 9
port priority: 255
port number: 2
port state: 69
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

 

[phastier]

By the way !
The redundancy works. If I unplug any of the ports, the bond acts normally, and partner lacp pdu information will appear on second interface. If I plug it back, the information disappears again.

All along, on the switch side, the lag link is considered as "down".

 

[phastier]

So.... the link is now up, the switch sees it as active, and I can see PDUs on the host side. I'm not sure why it now works to be honest. :/ I rebooted the switch, but only one of the six members now shous 802.3ad as up.

However, I can SSH from any host except from any member of the cluster (the SSH just lags), and the web interface is non working. In fact, even SSH sessions do not seem reliable.

If I do a pvecm status, the cluster looks just fine.
If I do an nmap, port 8006 is still open normally.

Did I miss any reconfiguration I should do now that the lag is "properly" 802.3ad ?

 

[phastier]

OK ! I nailed it.

I was honestly not lucky :
- 2 non-consecutive ports were failing on the switch (I mean physically...),
- some old VLAN configuration also remained on the switch,
- some ports had LACP tuned off !
- and by default, new trunks did not use jumbo frames on the switch.....

So all issues were indeed on switch side....

Now I have all my 6 cluster members properly using 802.3ad with jumbo frames. Good !

Now on to finishing the CEPH cluster configuration