Hello!
We have an issue with our PVE deployment. When a server is booting up, it causes a brief (20 seconds) broadcast storm.
We configured a LACP bond, but we also had this same issue with an active-backup bond previously. The bond is configured to default management bridge vmbr1 that has a management IP configured for web interface and cluster communication.
Overview of used HW:
Lenovo SR630 type 7X02 with X722 LOM NIC (4 optical 10Gb interfaces)
2x HPE FlexFabric 5945 48SFP28 8QSFP28 Switch (JQ074A) (configured in stack)
eno1np0 is connected to Twenty-FiveGigE1/0/23
eno3np2 is connected to Twenty-FiveGigE2/0/23 (slave switch)
port speed is manually set to 10Gb on the switches for it to create a link in 25Gb ports
The servers have latest firmware updates installed.
Here are relevant configurations:
Installed packages:
Once the server is up, the bond behaves as expected (in this example I shut one of the ports administratively and brought it up again, then I tested it on the other port in bond and it also worked as expected):
We have two additional 25Gb NICs installed and used but they don't have any bonds configured and they are in different VLANs. We're sure they are not causing this issue as we had this issue before configuring them. The first thing we set up after a fresh install was this bond and the problem started happening after configuring it. If we boot the server up with one of the interfaces administratively down on the switch the broadcast storm doesn't happen. We're deducting that it's a broadcast storm from looking at interface statistics. This behavior is consistent accross all three of our servers we have in deployment right now.
We think a relevant clue is that the storm happens for 20 seconds. The only timer on the switch that's so fast is STP hello, but we ran out of theories why and how this can happen.
What we didn't try is turning on stp on the management bridge. We also don't have fast mode turned on on the bond. We'll test if those will mitigate the issue outside usual business hours to not affect our customers.
It would be very helpful to us if you had similar experience and found a solution or if you have any ideas for where to look. I can also provide additional details if needed.
Thank you and have a nice rest of the day!
We have an issue with our PVE deployment. When a server is booting up, it causes a brief (20 seconds) broadcast storm.
We configured a LACP bond, but we also had this same issue with an active-backup bond previously. The bond is configured to default management bridge vmbr1 that has a management IP configured for web interface and cluster communication.
Overview of used HW:
Lenovo SR630 type 7X02 with X722 LOM NIC (4 optical 10Gb interfaces)
2x HPE FlexFabric 5945 48SFP28 8QSFP28 Switch (JQ074A) (configured in stack)
eno1np0 is connected to Twenty-FiveGigE1/0/23
eno3np2 is connected to Twenty-FiveGigE2/0/23 (slave switch)
port speed is manually set to 10Gb on the switches for it to create a link in 25Gb ports
The servers have latest firmware updates installed.
Here are relevant configurations:
Code:
root@nextclouda:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
auto eno1np0
iface eno1np0 inet manual
iface ens1f0np0 inet manual
#WAN
auto ens1f1np1
iface ens1f1np1 inet static
address 10.0.0.1/24
mtu 9000
#ceph-cluster
auto eno2np1
iface eno2np1 inet static
address 10.0.1.1/24
mtu 9000
#ceph-public
auto eno3np2
iface eno3np2 inet manual
iface eno4np3 inet manual
iface ens2f0np0 inet manual
iface ens2f1np1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves eno1np0 eno3np2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#MNG
auto vmbr0
iface vmbr0 inet static
address 10.70.68.190/22
gateway 10.70.68.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
#MNG
source /etc/network/interfaces.d/*
Code:
root@nextclouda:~# ethtool -i eno1np0
driver: i40e
version: 6.8.12-8-pve
firmware-version: 4.10 0x80001b6f 1.2203.0
expansion-rom-version:
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Code:
root@nextclouda:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.8.12-8-pve
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 38:68:dd:1c:2c:e8
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 15
Partner Key: 2
Partner Mac Address: 4c:ae:a3:ad:3f:f3
Slave Interface: eno1np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 38:68:dd:1c:2c:e8
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 38:68:dd:1c:2c:e8
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 32768
system mac address: 4c:ae:a3:ad:3f:f3
oper key: 2
port priority: 32768
port number: 3
port state: 61
Slave Interface: eno3np2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 38:68:dd:1c:2c:ea
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 38:68:dd:1c:2c:e8
port key: 15
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 32768
system mac address: 4c:ae:a3:ad:3f:f3
oper key: 2
port priority: 32768
port number: 4
port state: 61
Code:
<Switch>show int Twenty-FiveGigE1/0/23
Twenty-FiveGigE1/0/23
Current state: UP
Line protocol state: UP
IP packet frame type: Ethernet II, hardware address: 4cae-a3ad-403b
Description: Twenty-FiveGigE1/0/23 Interface
Bandwidth: 10000000 kbps
Loopback is not set
Media type is stack wire, port hardware type is STACK_SFP_PLUS
Ethernet port mode: LAN
10Gbps-speed mode, full-duplex mode
Link speed type is force link, link duplex type is autonegotiation
Flow-control is not enabled
Maximum frame length: 9416
Allow jumbo frames to pass
Broadcast max-ratio: 100%
Multicast max-ratio: 100%
Unicast max-ratio: 100%
PVID: 1
MDI type: Automdix
Port link-type: Access
Tagged VLANs: None
Untagged VLANs: 1
Port priority: 0
Last link flapping: 6 days 19 hours 22 minutes
Last clearing of counters: Never
Current system time:2025-03-28 08:52:55
Last time when physical state changed to up:2025-03-21 13:30:22
Last time when physical state changed to down:2025-03-21 13:30:17
Peak input rate: 131205607 bytes/sec, at 2025-03-11 10:15:05
Peak output rate: 102405387 bytes/sec, at 2025-03-24 07:37:59
Last 300 seconds input: 50 packets/sec 10980 bytes/sec 0%
Last 300 seconds output: 226 packets/sec 42176 bytes/sec 0%
Input (total): 94592418 packets, 73782102672 bytes
94532436 unicasts, 12620 broadcasts, 47362 multicasts, 0 pauses
Input (normal): 94592418 packets, - bytes
94532436 unicasts, 12620 broadcasts, 47362 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 214636571 packets, 167398757726 bytes
205208359 unicasts, 7535339 broadcasts, 1892873 multicasts, 0 pauses
Output (normal): 214636571 packets, - bytes
205208359 unicasts, 7535339 broadcasts, 1892873 multicasts, 0 pauses
Output: 0 output errors, - underruns, 0 buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
IPv4 traffic statistics:
Last 0 seconds input rate: 0 packets/sec, 0 bytes/sec
Last 0 seconds output rate: 0 packets/sec, 0 bytes/sec
Input: 0 packets, 0 bytes
Output: 0 packets, 0 bytes
IPv6 traffic statistics:
Last 0 seconds input rate: 0 packets/sec, 0 bytes/sec
Last 0 seconds output rate: 0 packets/sec, 0 bytes/sec
Input: 0 packets, 0 bytes
Output: 0 packets, 0 bytes
<Switch>show int Twenty-FiveGigE2/0/23
Twenty-FiveGigE2/0/23
Current state: UP
Line protocol state: UP
IP packet frame type: Ethernet II, hardware address: 4cae-a3ad-6f57
Description: Twenty-FiveGigE2/0/23 Interface
Bandwidth: 10000000 kbps
Loopback is not set
Media type is stack wire, port hardware type is STACK_SFP_PLUS
Ethernet port mode: LAN
10Gbps-speed mode, full-duplex mode
Link speed type is force link, link duplex type is autonegotiation
Flow-control is not enabled
Maximum frame length: 9416
Allow jumbo frames to pass
Broadcast max-ratio: 100%
Multicast max-ratio: 100%
Unicast max-ratio: 100%
PVID: 1
MDI type: Automdix
Port link-type: Access
Tagged VLANs: None
Untagged VLANs: 1
Port priority: 0
Last link flapping: 6 days 19 hours 22 minutes
Last clearing of counters: Never
Current system time:2025-03-28 08:53:08
Last time when physical state changed to up:2025-03-21 13:30:21
Last time when physical state changed to down:2025-03-21 13:30:17
Peak input rate: 5458221 bytes/sec, at 2025-03-24 13:22:36
Peak output rate: 279172873 bytes/sec, at 2025-03-24 07:59:53
Last 300 seconds input: 198 packets/sec 40843 bytes/sec 0%
Last 300 seconds output: 19 packets/sec 4446 bytes/sec 0%
Input (total): 82773085 packets, 17279694724 bytes
82538236 unicasts, 6193 broadcasts, 228656 multicasts, 0 pauses
Input (normal): 82773085 packets, - bytes
82538236 unicasts, 6193 broadcasts, 228656 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 136673541 packets, 136899148103 bytes
92004047 unicasts, 35147489 broadcasts, 9522005 multicasts, 0 pauses
Output (normal): 136673541 packets, - bytes
92004047 unicasts, 35147489 broadcasts, 9522005 multicasts, 0 pauses
Output: 0 output errors, - underruns, 0 buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
IPv4 traffic statistics:
Last 0 seconds input rate: 0 packets/sec, 0 bytes/sec
Last 0 seconds output rate: 0 packets/sec, 0 bytes/sec
Input: 0 packets, 0 bytes
Output: 0 packets, 0 bytes
IPv6 traffic statistics:
Last 0 seconds input rate: 0 packets/sec, 0 bytes/sec
Last 0 seconds output rate: 0 packets/sec, 0 bytes/sec
Input: 0 packets, 0 bytes
Output: 0 packets, 0 bytes
<Switch>show int bagg2
Bridge-Aggregation2
Current state: UP
Line protocol state: UP
IP packet frame type: Ethernet II, hardware address: 4cae-a3ad-403b
Description: Bridge-Aggregation2 Interface
Bandwidth: 20000000 kbps
20Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 1
Port link-type: Access
Tagged VLANs: None
Untagged VLANs: 1
Last clearing of counters: Never
Last 300 seconds input: 249 packets/sec 52087 bytes/sec 0%
Last 300 seconds output: 251 packets/sec 47099 bytes/sec 0%
Input (total): 107308801 packets, 24801372697 bytes
107227533 unicasts, 7052 broadcasts, 74216 multicasts, 0 pauses
Input (normal): 107308801 packets, - bytes
107227533 unicasts, 7052 broadcasts, 74216 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 248036291 packets, 247588577981 bytes
241612221 unicasts, 5148823 broadcasts, 1275247 multicasts, 0 pauses
Output (normal): 248036291 packets, - bytes
241612221 unicasts, 5148823 broadcasts, 1275247 multicasts, 0 pauses
Output: 0 output errors, - underruns, 0 buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<Switch>show stp interface bagg2
----[CIST][Port2149(Bridge-Aggregation2)][FORWARDING]----
Port protocol : Enabled
Port role : Designated Port
Port ID : 128.2149
Port cost(Dot1T) : Config=auto, Active=1000
Desg.bridge/port : 32768.4cae-a3ad-3ff3, 128.2149
Port edged : Config=disabled, Active=disabled
Point-to-Point : Config=auto, Active=true
Transmit limit : 10 packets/hello-time
TC-Restriction : Disabled
Role-Restriction : Disabled
Protection type : Config=none, Active=none
MST BPDU format : Config=auto, Active=802.1s
Port Config-
Digest-Snooping : Disabled
Rapid transition : False
Num of VLANs mapped : 1
Port times : Hello 2s MaxAge 20s FwdDelay 15s MsgAge 0s RemHops 20
BPDU sent : 294883
TCN: 0, Config: 0, RST: 0, MST: 294883
BPDU received : 0
TCN: 0, Config: 0, RST: 0, MST: 0Installed packages:
Code:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-9-pve)
pve-manager: 8.3.5 (running version: 8.3.5/dac3aa88bac3f300)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8: 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph: 19.2.0-pve2
ceph-fuse: 19.2.0-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
dnsmasq: 2.90-4~deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.2.0
libpve-network-perl: 0.10.1
libpve-rs-perl: 0.9.2
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.4-1
proxmox-backup-file-restore: 3.3.4-1
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.7
pve-cluster: 8.0.10
pve-container: 5.2.4
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.4.1
pve-qemu-kvm: 9.2.0-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2Once the server is up, the bond behaves as expected (in this example I shut one of the ports administratively and brought it up again, then I tested it on the other port in bond and it also worked as expected):
Code:
port WGE1/0/23 shut:
[ 405.231739] i40e 0000:09:00.0 eno1np0: NIC Link is Down
[ 405.491703] bond0: (slave eno1np0): speed changed to 0 on port 1
[ 405.493307] bond0: (slave eno1np0): link status definitely down, disabling slave
[ 405.498717] i40e 0000:09:00.0: entering allmulti mode.
port turned on:
[ 451.594671] i40e 0000:09:00.0 eno1np0: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 451.670246] bond0: (slave eno1np0): link status definitely up, 10000 Mbps full duplex
[ 451.670279] bond0: active interface up!We have two additional 25Gb NICs installed and used but they don't have any bonds configured and they are in different VLANs. We're sure they are not causing this issue as we had this issue before configuring them. The first thing we set up after a fresh install was this bond and the problem started happening after configuring it. If we boot the server up with one of the interfaces administratively down on the switch the broadcast storm doesn't happen. We're deducting that it's a broadcast storm from looking at interface statistics. This behavior is consistent accross all three of our servers we have in deployment right now.
Code:
(23s caused storm, 24s didn't)
<Switch>show counters inbound int Twenty-FiveGigE1/0/24
Interface Total (pkts) Broadcast (pkts) Multicast (pkts) Err (pkts)
WGE1/0/24 256022910 96128674 159886421 0
<Switch>show counters inbound int Twenty-FiveGigE2/0/24
Interface Total (pkts) Broadcast (pkts) Multicast (pkts) Err (pkts)
WGE2/0/24 2474403051 590275472 1877378319 0
<Switch>show counters inbound int Twenty-FiveGigE1/0/23
Interface Total (pkts) Broadcast (pkts) Multicast (pkts) Err (pkts)
WGE1/0/23 69866969 11763 2577 0
<Switch>show counters inbound int Twenty-FiveGigE2/0/23
Interface Total (pkts) Broadcast (pkts) Multicast (pkts) Err (pkts)
WGE2/0/23 205166 11 203946 0We think a relevant clue is that the storm happens for 20 seconds. The only timer on the switch that's so fast is STP hello, but we ran out of theories why and how this can happen.
What we didn't try is turning on stp on the management bridge. We also don't have fast mode turned on on the bond. We'll test if those will mitigate the issue outside usual business hours to not affect our customers.
It would be very helpful to us if you had similar experience and found a solution or if you have any ideas for where to look. I can also provide additional details if needed.
Thank you and have a nice rest of the day!