LACP issue with CEPH cluster and Cisco 9k Switch

J

jleubank

New Member
Aug 16, 2025
7
0
1
I've been working with a vendor of ours to help speed up a cluster deployment however we've been running into a wall with the LAGs forming. I left Corosync out of the mix below since there is no issue there seeing its in Active/Standby. But after setup for the Ceph and VM network we found that only one node would form the LAG at a time where as the others wouldn't not send LACP PDUs from the Proxmox nodes. This has been confirmed via the Cisco switch logs and the ports going into a suspension status because of the lack of response from the other three nodes. The linux configuration from I saw during the session with the vendor is matched between each Proxmox node and the example of how they have it configured is below which was pulled from the admin guide. Now for the LAG formation behaviors we seen. Node 1 forms bundle, node 2-4 does not, restart the network services on all for nodes in linux then node 1 stops, node 2 forms, 3-4 never forms. As another test I bounce the port channel instead and then node 4 forms yet nodes 1-3 never form. I've never seen anything like this and I am at a loss.

Layer 1
Cisco
Te1/0/1, Te1/0/2, Te1/0/5, Te1/0/6, Te1/0/9, Te1/0/10, Te1/0/13, Te1/0/14, Po100 (VM Network)

Te1/0/3, Te1/0/4, Te1/0/7, Te1/0/8, Te1/0/11, Te1/0/12, Te1/0/15, Te1/0/16, Po103 (Ceph/Storage Network)

Proxmox 4-Node Cluster
Node 1
NIC 1 Port 1, NIC 2 Port 1 - Bond 0 (VM Network)
NIC 1 Port 2, NIC 2 Port 2 - Bond 1 (Ceph/Storage Network)

Node 2
NIC 1 Port 1, NIC 2 Port 1 - Bond 0 (VM Network)
NIC 1 Port 2, NIC 2 Port 2 - Bond 1 (Ceph/Storage Network)

Node 3
NIC 1 Port 1, NIC 2 Port 1 - Bond 0 (VM Network)
NIC 1 Port 2, NIC 2 Port 2 - Bond 1 (Ceph/Storage Network)

Node 4
NIC 1 Port 1, NIC 2 Port 1 - Bond 0 (VM Network)
NIC 1 Port 2, NIC 2 Port 2 - Bond 1 (Ceph/Storage Network)

Layer 2:
Cisco

Port-Channel100
description BLAH
switchport mode trunk
switchport trunk allowed vlan add vlan 100,199

Port-Channel103
description BLAH
switchport mode access
switchport access vlan 103

interface TenGigEthernet1/0/1,2,5,6,9,10,13,14
description BLAH
switchport mode trunk
switchport trunk allowed vlan add 100,199
channel-group 100 mode active

interface TenGigEthernet1/0/3,4,7,8,11,12,15,16
description BLAH
switchport mode access
switchport access vlan 103
channel-group 103 mode active

Proxmox 4-Node Custer (Mind you I can't get into it currently due to credential handoff from vendor but this is how I know its configured)
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 10.10.10.2/24
gateway 10.10.10.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
 
Do you have LACP rate set to fast or slow on the switch? It needs to match on the Linux side too. The default is slow on Linux.

I don’t know where you got eno.. from but there is an alt name that describes port numbers as well, it may be better to use those to make sure you’re putting the right ports into the bond, you should have 2 p1 and 2 p2 in the bond given your explanation.

You also seem to put all ports in 1 channel on the switch side, you should make a channel of the pairs for each node.
 
Last edited:
As for the ports I do know they line up. This was confirmed by breaking the bonds and running interface to interface then running the mac table on the switch to line everything up since I was remote. This is how I knew the vendor had the ports mixed up. Your thoughts and mine were the same and had to lay out for them 1+1=2. This was corrected prior to the weird LACP behaviors. Give me a bit and I will get in to see what the rate is. Cisco is typically slow at 30 seconds. But I will confirm and see what I am pulling from the single neighbor on their rate. As far as your last statement we have to port channels, 100 which is for VM and 103 for Ceph. Even on the Corosync that too is another channel. Layer 2 breakdown is this for each:

VM
Port-Channel100
Vlan 100-199

Storage
Port-Channel103
Vlan 103

Corosync
AG2
Vlan 104
 
jleubank said:
Layer 2:
Cisco

Port-Channel100
description BLAH
switchport mode trunk
switchport trunk allowed vlan add vlan 100,199

Port-Channel103
description BLAH
switchport mode access
switchport access vlan 103

interface TenGigEthernet1/0/1,2,5,6,9,10,13,14
description BLAH
switchport mode trunk
switchport trunk allowed vlan add 100,199
channel-group 100 mode active

interface TenGigEthernet1/0/3,4,7,8,11,12,15,16
description BLAH
switchport mode access
switchport access vlan 103
channel-group 103 mode active
Click to expand...
At the CISCO side, it should have each port-channel includes 2 TenGigPorts and connect to single Proxmox VE Server. So if you have 4 Proxmox VE nodes, you should have 4 port-channels on the CISCO Switch, each port-channel connect with single Proxmox VE node.

Just FYR.
 
Here is the data I pulled:
Switch Config:

!

interface Port-channel100

description NNI 80G P2P wcg-psb-proxmox cluster1 via Te1/1&2/1

switchport trunk allowed vlan 100,199

switchport mode trunk

!

interface Port-channel103

description NNI 80G P2P wcg-psb-proxmox cluster1 via Te1/2&2/2

switchport access vlan 103

switchport mode access

!

interface TenGigabitEthernet1/0/1

description NNI 10G P2P wcg-psb-prox1 ETH1/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/2

description NNI 10G P2P wcg-psb-prox1 ETH2/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/3

description NNI 10G P2P wcg-psb-prox1 ETH1/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/4

description NNI 10G P2P wcg-psb-prox1 ETH2/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/5

description NNI 10G P2P wcg-psb-prox2 ETH1/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/6

description NNI 10G P2P wcg-psb-prox2 ETH2/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/7

description NNI 10G P2P wcg-psb-prox2 ETH1/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/8

description NNI 10G P2P wcg-psb-prox2 ETH2/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/9

description NNI 10G P2P wcg-psb-prox3 ETH1/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/10

description NNI 10G P2P wcg-psb-prox3 ETH2/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/11

description NNI 10G P2P wcg-psb-prox3 ETH1/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/12

description NNI 10G P2P wcg-psb-prox3 ETH2/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/13

description NNI 10G P2P wcg-psb-prox4 ETH1/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/14

description NNI 10G P2P wcg-psb-prox4 ETH2/1

switchport trunk allowed vlan 100,199

switchport mode trunk

channel-group 100 mode active

lacp rate fast

!

interface TenGigabitEthernet1/0/15

description NNI 10G P2P wcg-psb-prox4 ETH1/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!

interface TenGigabitEthernet1/0/16

description NNI 10G P2P wcg-psb-prox4 ETH2/2

switchport access vlan 103

switchport mode access

channel-group 103 mode active

!



LACP Status:



Flags: S - Device is requesting Slow LACPDUs

F - Device is requesting Fast LACPDUs

A - Device is in Active mode P - Device is in Passive mode



Channel-Group 100 neighbors



LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Te1/0/1 FA 255 8416.0cc3.f940 0s 0x0 0xF 0x1 0x3F

Te1/0/2 FA 255 8416.0cc3.f940 0s 0x0 0xF 0x2 0x3F

Te1/0/5 FA 255 8416.0ccd.9be0 2s 0x0 0xF 0x1 0xF

Te1/0/6 FA 255 8416.0ccd.9be0 2s 0x0 0xF 0x2 0xF

Te1/0/9 FA 255 8416.0cc5.2c20 2s 0x0 0xF 0x1 0xF

Te1/0/10 FA 255 8416.0cc5.2c20 2s 0x0 0xF 0x2 0xF



Log Entries

Lag that is up:

Aug 16 14:20:46.841: LACP :lacp_bugpak: Receive LACP-PDU packet via Te1/0/2

Aug 16 14:20:46.842: LACP: Te1/0/2 LACP packet received, processing

Aug 16 14:20:46.842: lacp_rx Te1/0/2 - rx: during state CURRENT, got event 5(recv_lacpdu)

Aug 16 14:20:46.842: @@@ lacp_rx Te1/0/2 - rx: CURRENT -> CURRENT

Aug 16 14:20:46.842: LACP: Te1/0/2 lacp_action_rx_current entered

Aug 16 14:20:46.842: LACP: recordPDU Te1/0/2 LACP PDU Rcvd. Partners oper state is hex 3F

Aug 16 14:20:46.842: LACP: recordPDU Te1/0/2 Partner in sync and aggregating

Aug 16 14:20:46.842: LACP: Te1/0/2 Partners oper state is hex 3F

Aug 16 14:20:46.842: LACP: timer lacp_c_s(Te1/0/2) started with interval 3000.

Aug 16 14:20:46.842: LACP: Te1/0/2 LAG_PARTNER_UP.



LAG that is down:

Aug 16 14:21:26.964: LACP :lacp_bugpak: Send LACP-PDU packet via Te1/0/5

Aug 16 14:21:26.964: LACP: lacp_write: LACP 124 bytes out Te1/0/5

Aug 16 14:21:26.964: lacp_ptx Te1/0/5 - ptx: during state PERIODIC_TX, got event 4(short_timeout)

Aug 16 14:21:26.964: @@@ lacp_ptx Te1/0/5 - ptx: PERIODIC_TX -> FAST_PERIODIC

Aug 16 14:21:26.964: LACP: Te1/0/5 lacp_action_ptx_fast_periodic entered

Aug 16 14:21:26.964: LACP: timer lacp_p_f(Te1/0/5) started with interval 1000.

Aug 16 14:21:27.920: LACP: lacp_t(Te1/0/5) timer stopped

Aug 16 14:21:27.920: LACP: lacp_t(Te1/0/5) expired

Aug 16 14:21:27.952: LACP: lacp_p(Te1/0/5) timer stopped

Aug 16 14:21:27.952: LACP: lacp_p(Te1/0/5) expired


As far as your statement on your saying I can't run a single port channel say for example the VM network where it lags on the Cisco side all 8 interfaces spanning across the 4 nodes? If that would be accurate then that would be 12 port channels in total 3 per node.
 
Channel group 2 neighbors



LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Te1/0/5 FA 255 8416.0ccd.9be0 0s 0x0 0xF 0x1 0x3F

Te1/0/6 FA 255 8416.0ccd.9be0 1s 0x0 0xF 0x2 0x3F



Channel group 3 neighbors



LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Te1/0/9 FA 255 8416.0cc5.2c20 0s 0x0 0xF 0x1 0x3F

Te1/0/10 FA 255 8416.0cc5.2c20 0s 0x0 0xF 0x2 0x3F



Channel group 100 neighbors



LACP port Admin Oper Port Port

Port Flags Priority Dev ID Age key Key Number State

Te1/0/1 FA 255 8416.0cc3.f940 0s 0x0 0xF 0x1 0x3F

Te1/0/2 FA 255 8416.0cc3.f940 0s 0x0 0xF 0x2 0x3F
 
Bonds formed. Weird you are unable to form all interfaces in the cluster as a single 8 interface bond, that it wants it on a per node basis. Never seen that before.
 
Your LACP rate seems to be set to fast too, it will negotiate but a down link may not quickly if the rates don’t match.

A bond is not a VLAN, different layers, there is a mode you can set on some switches where it will just accept ‘any’ LACP message and accept it. So you just set all ports to accept LACP packets and then your switch will auto-negotiate them, that may be where you are confused, not sure how Cisco does that, but it is possible.
 
@guruevi Your talking about passive mode regarding LACP negotiations. I would have to see if it will affect the port channel and let you now.
 
You must log in or register to reply here.
Top Bottom