LACP issue on PVE 9.0.3

B

bandzior

Renowned Member
Jun 2, 2015
32
0
71
Hello.

I have an issue with LACP enabled connection with two physical Ethernet cards, which means that every time I restart the connection using the command:

Code: 
systemctl restart networking.service

It takes 30 seconds to reconnect. In dmesg after every restart I see the following logs:

Code: 
[Tue Aug 19 12:17:43 2025] vmbr2050: port 1(bond0.2050) entered disabled state
[Tue Aug 19 12:17:43 2025] bond0.2050: left allmulticast mode
[Tue Aug 19 12:17:43 2025] bond0.2050: left promiscuous mode
[Tue Aug 19 12:17:43 2025] vmbr2050: port 1(bond0.2050) entered disabled state
[Tue Aug 19 12:17:44 2025] vmbr2100: port 1(bond0.2100) entered disabled state
[Tue Aug 19 12:17:44 2025] bond0.2100: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0.2100: left promiscuous mode
[Tue Aug 19 12:17:44 2025] vmbr2100: port 1(bond0.2100) entered disabled state
[Tue Aug 19 12:17:44 2025] vmbr2110: port 1(bond0.2110) entered disabled state
[Tue Aug 19 12:17:44 2025] bond0.2110: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0.2110: left promiscuous mode
[Tue Aug 19 12:17:44 2025] vmbr2110: port 1(bond0.2110) entered disabled state
[Tue Aug 19 12:17:44 2025] vmbr2120: port 1(bond0.2120) entered disabled state
[Tue Aug 19 12:17:44 2025] bond0.2120: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0.2120: left promiscuous mode
[Tue Aug 19 12:17:44 2025] vmbr2120: port 1(bond0.2120) entered disabled state
[Tue Aug 19 12:17:44 2025] vmbr2140: port 1(bond0.2140) entered disabled state
[Tue Aug 19 12:17:44 2025] bond0.2140: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0.2140: left promiscuous mode
[Tue Aug 19 12:17:44 2025] vmbr2140: port 1(bond0.2140) entered disabled state
[Tue Aug 19 12:17:44 2025] vmbr2150: port 1(bond0.2150) entered disabled state
[Tue Aug 19 12:17:44 2025] bond0.2150: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0: left allmulticast mode
[Tue Aug 19 12:17:44 2025] i40e 0000:12:00.0 ens1f0np0: left allmulticast mode
[Tue Aug 19 12:17:44 2025] i40e 0000:5d:00.0 eno5np0: left allmulticast mode
[Tue Aug 19 12:17:44 2025] bond0.2150: left promiscuous mode
[Tue Aug 19 12:17:44 2025] bond0: left promiscuous mode
[Tue Aug 19 12:17:44 2025] i40e 0000:12:00.0 ens1f0np0: left promiscuous mode
[Tue Aug 19 12:17:44 2025] i40e 0000:5d:00.0 eno5np0: left promiscuous mode
[Tue Aug 19 12:17:44 2025] vmbr2150: port 1(bond0.2150) entered disabled state
[Tue Aug 19 12:17:44 2025] i40e 0000:5d:00.0: leaving allmulti mode.
[Tue Aug 19 12:17:44 2025] i40e 0000:12:00.0: leaving allmulti mode.
[Tue Aug 19 12:17:44 2025] bond0 (unregistering): (slave ens1f0np0): Releasing backup interface
[Tue Aug 19 12:17:44 2025] bond0 (unregistering): (slave eno5np0): Removing an active aggregator
[Tue Aug 19 12:17:44 2025] bond0 (unregistering): (slave eno5np0): Releasing backup interface
[Tue Aug 19 12:17:44 2025] i40e 0000:5d:00.0 eno5np0: returning to hw mac address d4:f5:ef:20:f1:70
[Tue Aug 19 12:17:44 2025] bond0 (unregistering): Released all slaves
[Tue Aug 19 12:17:45 2025] 8021q: adding VLAN 0 to HW filter on device ens1f1np1
[Tue Aug 19 12:17:45 2025] 8021q: adding VLAN 0 to HW filter on device eno6np1
[Tue Aug 19 12:17:45 2025] 8021q: adding VLAN 0 to HW filter on device ens1f0np0
[Tue Aug 19 12:17:45 2025] bond0: (slave ens1f0np0): Enslaving as a backup interface with an up link
[Tue Aug 19 12:17:45 2025] i40e 0000:5d:00.0 eno5np0: set new mac address d4:f5:ef:31:22:c0
[Tue Aug 19 12:17:45 2025] 8021q: adding VLAN 0 to HW filter on device eno5np0
[Tue Aug 19 12:17:45 2025] bond0: (slave eno5np0): Enslaving as a backup interface with an up link
[Tue Aug 19 12:17:45 2025] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[Tue Aug 19 12:17:45 2025] 8021q: adding VLAN 0 to HW filter on device bond0
[Tue Aug 19 12:17:45 2025] vmbr2050: port 1(bond0.2050) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2050: port 1(bond0.2050) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2050: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] i40e 0000:12:00.0 ens1f0np0: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] i40e 0000:5d:00.0 eno5np0: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2050: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] bond0: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] i40e 0000:12:00.0 ens1f0np0: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] i40e 0000:5d:00.0 eno5np0: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2050: port 1(bond0.2050) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2050: port 1(bond0.2050) entered forwarding state
[Tue Aug 19 12:17:45 2025] i40e 0000:5d:00.0: entering allmulti mode.
[Tue Aug 19 12:17:45 2025] i40e 0000:12:00.0: entering allmulti mode.
[Tue Aug 19 12:17:45 2025] vmbr2100: port 1(bond0.2100) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2100: port 1(bond0.2100) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2100: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2100: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2100: port 1(bond0.2100) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2100: port 1(bond0.2100) entered forwarding state
[Tue Aug 19 12:17:45 2025] vmbr2110: port 1(bond0.2110) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2110: port 1(bond0.2110) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2110: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2110: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2110: port 1(bond0.2110) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2110: port 1(bond0.2110) entered forwarding state
[Tue Aug 19 12:17:45 2025] vmbr2120: port 1(bond0.2120) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2120: port 1(bond0.2120) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2120: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2120: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2120: port 1(bond0.2120) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2120: port 1(bond0.2120) entered forwarding state
[Tue Aug 19 12:17:45 2025] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[Tue Aug 19 12:17:45 2025] bond0: (slave ens1f0np0): link status definitely up, 10000 Mbps full duplex
[Tue Aug 19 12:17:45 2025] bond0: active interface up!
[Tue Aug 19 12:17:45 2025] vmbr2140: port 1(bond0.2140) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2140: port 1(bond0.2140) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2140: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2140: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2140: port 1(bond0.2140) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2140: port 1(bond0.2140) entered forwarding state
[Tue Aug 19 12:17:45 2025] vmbr2150: port 1(bond0.2150) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2150: port 1(bond0.2150) entered disabled state
[Tue Aug 19 12:17:45 2025] bond0.2150: entered allmulticast mode
[Tue Aug 19 12:17:45 2025] bond0.2150: entered promiscuous mode
[Tue Aug 19 12:17:45 2025] vmbr2150: port 1(bond0.2150) entered blocking state
[Tue Aug 19 12:17:45 2025] vmbr2150: port 1(bond0.2150) entered forwarding state
[Tue Aug 19 12:17:45 2025] bond0: (slave eno5np0): link status definitely up, 10000 Mbps full duplex

I checked the PCAPs from the server and found nothing alarming - no errors. On the other side are two Cisco Nexu 3000 series switches connected via vPC. LACP is in fast mode on both sides.

The LACP configuration on the server looks like this:

Code: 
auto bond0
iface bond0 inet manual
    bond-lacp-rate 1
    bond-slaves ens1f0np0 eno5np0
    bond-miimon 100
    bond-mode 802.3ad
    bond-downdelay 200
    bond-updelay 200
    bond-xmit-hash-policy layer2+3
    mtu 9000

# VLAN 2050 - MGMT PROX
auto vmbr2050
iface vmbr2050 inet static
    bridge-ports bond0.2050
    bridge-stp off
    bridge-fd 0
    address 10.40.50.11
    netmask 255.255.255.0
    gateway 10.40.50.1
    mtu 9000
    
# VLAN 2100 PROD
auto vmbr2100
iface vmbr2100 inet manual
    bridge-ports bond0.2100
    bridge-stp off
    bridge-fd 0
    mtu 9000

Do you have any ideas on how to solve this problem? This is the first time I've encountered something like this and I'm not quite sure how to solve it.

BTW in a bit, i will add configuration from Cisco Nexus switches.

Regards, Daniel
 
I had the exact same problem with our Mikrotik switches in a LACP/MLAG configuration. It was very unstable. I fixed it by changing the bond hash policy to layer2 only. I lost the load balancing I was getting before, but it is much more stable. Also, you could try using ifreload -a after configuration changes instead of restaring the networking services. This is how the Proxmox gui does it.
 
INI: 
ifreload -a
is the key.

I didn't know that command because I didn't have to use it before. BTW I didn't change anything in the configuration.

Thank you very much!
 
Last edited:
Good that you found a working solution.
(I would have asked why you would use the systemd-way, but you already found a better one)
 
You must log in or register to reply here.
Top Bottom