Bridge stops forwarding packets over the bond.

N

ninad

Member
May 12, 2023
4
0
6
Dear All,

I have a 5-node hyperconverged Proxmox cluster on a Dell R7525 with 2x2-port Intel E810 25G NICs. All these 5 nodes are connected to a stack of Extreme X695. Out of 4 ports, I have created bond0 with 2 ports and bond1 with the remaining 2 ports. The bond0 is used for data and proxmox management, and bond1 is used for cluster and Ceph communication. The setup is working fine until I updated the proxmox from 9.1.1 to 9.1.9
After the update, one port in bond0 is always in the attached state and does not go to Collect-Dist. As a result, all the VMs running on that host can only communicate with each other and proxmox host, not with the other VMs running on other nodes. Even the gateway is unreachable.
If I revert the setup (reinstall proxmox) to 9.1.1, everything works fine. Please let me know whether it's a bug or if I'm doing something wrong.

pveversion --verbose
proxmox-ve: 9.1.0 (running kernel: 6.17.2-1-pve)
pve-manager: 9.1.9 (running version: 9.1.9/ee7bad0a3d1546c9)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-7.0: 7.0.2-2
proxmox-kernel-7.0.2-2-pve-signed: 7.0.2-2
proxmox-kernel-6.17.13-7-pve-signed: 6.17.13-7
proxmox-kernel-6.17: 6.17.13-7
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
amd64-microcode: 3.20251202.1~bpo13+1
ceph: 19.2.3-pve4
ceph-fuse: 19.2.3-pve4
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx12
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.7
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.2
libpve-cluster-perl: 9.1.2
libpve-common-perl: 9.1.11
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.3.0
libpve-notify-perl: 9.1.2
libpve-rs-perl: 0.13.0
libpve-storage-perl: 9.1.2
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-4
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-4
proxmox-backup-client: 4.2.0-1
proxmox-backup-file-restore: 4.2.0-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.2
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.9
pve-cluster: 9.1.2
pve-container: 6.1.5
pve-docs: 9.1.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-3
pve-ha-manager: 5.2.0
pve-i18n: 3.7.1
pve-qemu-kvm: 10.1.2-7
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.9
smartmontools: 7.4-pve1
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.3.4-pve1



/etc/network/interface file:
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet static
address 10.205.a.b/24
#pxProxyClusLink0

iface eth5 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eth0 eth2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
#mainDataLink

auto bond1
iface bond1 inet static
address 10.205.x.y/24
bond-slaves eth1 eth3
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
#pxProxyClusLink1

auto bond0.200
iface bond0.200 inet manual
#200 NW VLAN

auto bond0.201
iface bond0.201 inet manual
#201 NW VLAN

auto bond0.203
iface bond0.203 inet manual
#203 NW VLAN

auto bond0.209
iface bond0.209 inet manual
#209 NW VLAN

auto vmbr200
iface vmbr200 inet static
address 10.200.x.y/16
gateway 10.200.250.1
bridge-ports bond0.200
bridge-stp off
bridge-fd 0
#200 NW

auto vmbr201
iface vmbr201 inet static
address 10.201.x.y/16
bridge-ports bond0.201
bridge-stp off
bridge-fd 0
#201 NW

auto vmbr203
iface vmbr203 inet static
address 10.203.x.y/16
bridge-ports bond0.203
bridge-stp off
bridge-fd 0
#203 NW

auto vmbr209
iface vmbr209 inet static
address 10.209.x.y/16
bridge-ports bond0.209
bridge-stp off
bridge-fd 0
#209 NW

#cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.17.2-1-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: b4:83:51:02:78:30
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 21
Partner Key: 7015
Partner Mac Address: f6:ce:48:f6:59:14

Slave Interface: eth0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b4:83:51:02:78:30
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: b4:83:51:02:78:30
port key: 21
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 0
system mac address: f6:ce:48:f6:59:14
oper key: 7015
port priority: 0
port number: 7015
port state: 61

Slave Interface: eth2
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b4:83:51:02:77:9c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: b4:83:51:02:78:30
port key: 21
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 0
system mac address: f6:ce:48:f6:59:14
oper key: 7015
port priority: 0
port number: 6031
port state: 61


Command used to configure a lag on a switch:

enable sharing 7:13 grouping 7:13 6:29 algorithm address-based L3_L4 distribution-mode all lacp

#show lacp lag 7:13 detail

Lag Actor Actor Partner Partner Partner Agg Actor
Sys-Pri Key MAC Sys-Pri Key Count MAC
--------------------------------------------------------------------------------
7:13 0 0x1b65 b4:83:51:02:cc:64 65535 0x0015 1 f6:ce:48:f6:59:14

Enabled : Yes
LAG State : Up
Unack count : 0
Wait-for-count : 0
Current timeout : Long
Activity mode : Active
Defaulted Action : Delete
Fallback : Disabled
Fallback timeout : 60 seconds
Receive state : Enabled
Transmit state : Enabled
Minimum active : 1
Selected count : 2
Standby count : 0
LAG Id flag : Yes
S.pri:0 , S.id:f6:ce:48:f6:59:14, K:0x1b65
T.pri:65535, T.id:b4:83:51:02:cc:64, L:0x0015

Port list:

Member Port Rx Sel Mux Actor Partner
Port Priority State Logic State Flags Port
--------------------------------------------------------------------------------
6:29 0 Current Selected Attached A-GS---- 2
7:13 0 Current Selected Collect-Dist A-GSCD-- 1
================================================================================
Actor Flags: A-Activity, T-Timeout, G-Aggregation, S-Synchronization
C-Collecting, D-Distributing, F-Defaulted, E-Expired
 
ninad said:
If I revert the setup (reinstall proxmox) to 9.1.1, everything works fine. Please let me know whether it's a bug or if I'm doing something wrong.
Click to expand...
I think you have answered your own question, ie it does look like a bug if you can revert a change and get it working again!

However that does assume that no other changes have happened reasonably recently. I use bonding quite a lot, generally using static LACP (ie I specify the available ports at both ends) and it just works once you get the hashing algos to match up.

I can't give specific debugging notes for you because I don't have your gear. You should be able to debug LACP at both ends and this is a really good time to engage a LLM. They really are a better search with nobs on for this sort of job. Double check that bond-xmit-hash-policy layer3+4 matches what your switch wants to do. I find that every implementation of LACP uses subtly differing terminology, despite it being a formal IEEEEEEeeeeeee standard.

Switch manuals generally (Dell/HPE/Cisco ... even Netgear!) have loads of advice on this sort of thing but I'm not familiar with Extreme gear.
 
@Blueloop Thank you for the reply.

One more observation, Proxmox 9.1.1 comes with the 6.17.2-1-pve kernel. after the update to Proxmox 9.1.9, 2 kernel (6.17.13-7-pve and 7.0.2-2-pve) gets installed. If I boot the server with any of the new kernels, one port on bond0 is always in the attached state and does not go to Collect-Dist. However, if I boot the same server with the old 6.17.2.1 kernel, everything works fine without any configuration change.
 
You must log in or register to reply here.
Top Bottom
Back