Hi everyone,
After upgrading to Proxmox 9, I encountered a strange performance problem in my 3-node OpenFabric cluster. Each node has two 10Gb NICs: an Intel X520 and a Broadcom BCM57810, bonded with LACP. The nodes form a triangular full mesh with VXLAN flood-and-learn overlay.
What I saw:
Diagnosis:
Remembering old Broadcom hardware offload issues from two decades ago nightmare, I suspected offloading bugs in the Broadcom NICs causing encapsulated VXLAN traffic to degrade.
Fix:
I disabled several offloading features on Broadcom interfaces with:
Don't forget to make it persistent in
Result:
If you’re running Proxmox with Broadcom BCM57810 NICs and see asymmetric VXLAN throughput, try disabling hardware offloading on those NICs. It solved a frustrating problem for me and might help others too.
Hope this helps!
After upgrading to Proxmox 9, I encountered a strange performance problem in my 3-node OpenFabric cluster. Each node has two 10Gb NICs: an Intel X520 and a Broadcom BCM57810, bonded with LACP. The nodes form a triangular full mesh with VXLAN flood-and-learn overlay.
What I saw:
- Bare IP tests (using iperf) ran perfectly at ~9.8Gbps both ways, as expected for LACP bonded links.
- However, when testing IPs inside VXLAN, one direction (Broadcom → Intel) bandwidth dropped drastically to around 13Mbps, while the other direction stayed at ~9.8Gbps.
Diagnosis:
Remembering old Broadcom hardware offload issues from two decades ago nightmare, I suspected offloading bugs in the Broadcom NICs causing encapsulated VXLAN traffic to degrade.
Fix:
I disabled several offloading features on Broadcom interfaces with:
Bash:
for IF in enp132s0f0 enp132s0f1; do ethtool -K $IF tso off rx-checksumming off tx-checksumming off lro off rx-vlan-offload off tx-vlan-offload off; doneDon't forget to make it persistent in
/etc/network/interfaces by adding for all impacted interfaces:
Code:
post-up ethtool -K $IFACE tso off rx-checksumming off tx-checksumming off lro off rx-vlan-offload off tx-vlan-offload offResult:
- VXLAN bandwidth restored to ~9.8Gbps in both directions.
- VM hard drive latency dropped from several seconds to normal levels.
If you’re running Proxmox with Broadcom BCM57810 NICs and see asymmetric VXLAN throughput, try disabling hardware offloading on those NICs. It solved a frustrating problem for me and might help others too.
Hope this helps!
Last edited:
