No 802.3ad response

A

Airw0lf

Member
Apr 11, 2021
68
3
13
61
it-visibility.net
Team,

I have a Proxmox server with 2 network cards and bonding with lacp/802.3ad.

The syslog says there is no 802.3ad response:
Code: 
Jan 15 17:38:30 the-neb systemd-networkd[698]: bond0: Gained carrier
Jan 15 17:38:30 the-neb systemd-timesyncd[941]: Network configuration changed, trying to establish connection.
Jan 15 17:38:30 the-neb systemd-networkd[698]: vmbr0: Gained carrier
Jan 15 17:38:30 the-neb systemd-timesyncd[941]: Network configuration changed, trying to establish connection.
Jan 15 17:38:30 the-neb kernel: [   98.699509] bond0: (slave enp10s0): link status definitely up, 1000 Mbps full duplex
Jan 15 17:38:30 the-neb kernel: [   98.699514] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
Jan 15 17:38:30 the-neb kernel: [   98.699720] bond0: (slave enp5s0f0): link status definitely up, 1000 Mbps full duplex
Jan 15 17:38:30 the-neb kernel: [   98.699722] bond0: active interface up!
Jan 15 17:38:30 the-neb kernel: [   98.699733] vmbr0: port 1(bond0) entered blocking state
Jan 15 17:38:30 the-neb kernel: [   98.699737] vmbr0: port 1(bond0) entered forwarding state

However the log also says the bond0 port is up and forwarding (and is working as expected).
In addition the lacp status also says the link is up:
Code: 
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.8.12-5-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: c8:7f:54:70:03:a3
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 1
        Actor Key: 9
        Partner Key: 1102
        Partner Mac Address: c0:c9:e3:0e:7b:bf

The interface settings:
Code: 
auto enp10s0
iface enp10s0 inet manual
        bond-primary enp10s0
        bond-master bond0

auto enp5s0f0
iface enp5s0f0 inet manual
        bond-master bond0

iface enp5s0f1 inet manual

iface enp6s0f0 inet manual

iface enp6s0f1 inet manual

iface enp11s0f0 inet manual

iface enp11s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp10s0 enp5s0f0
        bond-mode 4
        bond-miimon 0
# Mode 4 - 802.3ad LACP loadbalancing
# Mode 5 - adaptive loadbalancing
# Mode 6 - geeft performance problemen.

auto vmbr0
iface vmbr0 inet static
        address 192.168.139.252/24
        gateway 192.168.139.241
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-400
        dns-domain tech.lan
# Main interface

The switch ports are configured with passive lacp since the lacp status says "lacp active: on"

Removing the miimon setting doesn't change this.

What am I overlooking?

With warm regards - Will
 
Last edited:
packetmonkey said:
It is generally recommended to leave your switch in active mode over passive mode. It is better to have both nodes trying to be active than both trying to be passive lol.
Click to expand...

Well... the same applies for active/active mode - both are trying to get things started by shouting at each other. Who is then supposed to be listening to either of the shoutng?
 
Last edited:
bbgeek17 said:
Hi Will,
Generally, we recommend using Fast LACP.

For your troubleshooting I would recommend doing a capture on the underlying physical interfaces:
tcpdump -ni <IFNAME> ether proto 0x8809

Are LACP packets flowing back and forth?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

Thank you for the quick response.

I tried capturing for about 10-15 seconds on the two actual ethernet ports, bond0 and vmbr0 - none of these are showing anything - captured packets are all 0.

The switch states hashing is based on "source mac and destination mac". Could this have anything to do with it? What would be the equivalent on Proxmox site? And how do I activate this?
 
Last edited:
Airw0lf said:
I tried capturing for about 10-15 seconds on the two actual ethernet ports, bond0 and vmbr0 - none of these are showing anything - captured packets are all 0.
Click to expand...
"Slow" LACP packets are sent every 30s, "Fast" are every second. Always use "fast"

Have you talked to your Network admin? While the host side config looks ok, may be your switch is misconfigured?

Start from the basics. Remove all the configuration on your host, bring the link up, and start network trace on raw ports. If you don't see any LACP packets sent by the switch (1-2min) - then your switch config is not what you think it is.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Airw0lf
Airw0lf said:
Well... the same applies for active/active mode - both are trying to get things started by shouting at each other. Who is then supposed to be listening to either of the shoutng?
Click to expand...
Have you seen what happens when both nodes are set to passive? If you intentionally and knowingly tell one side to be active and one to be passive, and it works, that is best. Outside of that, leave switches in active mode and let them do the negotiating. How much "shouting" do you think occurs outside of the negotiation packets?
 
  • Like
Reactions: Airw0lf and bbgeek17
packetmonkey said:
Have you seen what happens when both nodes are set to passive? If you intentionally and knowingly tell one side to be active and one to be passive, and it works, that is best. Outside of that, leave switches in active mode and let them do the negotiating. How much "shouting" do you think occurs outside of the negotiation packets?
Click to expand...

bbgeek17 said:
"Slow" LACP packets are sent every 30s, "Fast" are every second. Always use "fast"

Have you talked to your Network admin? While the host side config looks ok, may be your switch is misconfigured?

Start from the basics. Remove all the configuration on your host, bring the link up, and start network trace on raw ports. If you don't see any LACP packets sent by the switch (1-2min) - then your switch config is not what you think it is.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

I'm also the temp network admin - sort-of... :rolleyes:

I have no idea - not that much experience with LACP/802.3ad.
Usually I just do an upgrade to 10-Gbps ports in active/standby mode (versus loadbalancing via 2 or more 1-Gbps ports).

I did some reading on how this LACP/802.3ad is supposed to work and then looked for the factory defaults on the switch and the server.

It appears that the server has its defaults on active-LACP with L2-hashing.
It seems that the switch has no defaults => set it to passive-LACP and the hashes with "Source-MAC and destination-MAC"; assuming its the equivalent with the L2-hashes default on the server side (to-be confirmed by the switch vendor).
 
Last edited:
Airw0lf said:
I'm also the temp network admin - sort-of...
Click to expand...
Get the LACP packets flowing on the ports.

We always say that "the hardest part of the storage is networking". The number of times we were told that network/lacp are all configured and ready to go, only to find out that wrong ports were bonded on the switch side, is... large.

There is not much to do on the hypervisor side until you are 100% sure that switch is correctly configured.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Get the LACP packets flowing on the ports.

We always say that "the hardest part of the storage is networking". The number of times we were told that network/lacp are all configured and ready to go, only to find out that wrong ports were bonded on the switch side, is... large.

There is not much to do on the hypervisor side until you are 100% sure that switch is correctly configured.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

Well... I agree with your view about storage and networking.

However... if the mac-table of the switch is showing the mac-addresses of the 2 server network cards and both are presented with the correct port numbers, then I guess everything is connected (configured?) as expected - unless I'm overlooking something...

With that in mind - if I still don't see any LACP packets, then I guess something is not working as expected.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom