Bond setup

aasami · Jul 21, 2023

Hi! I have some troubles setting up bond on BCM57504 via S5224F-ON.
Network configuration:

Code:

auto lo
iface lo inet loopback

auto enp129s0f0np0
iface enp129s0f0np0 inet manual

auto enp129s0f1np1
iface enp129s0f1np1 inet manual

auto enp129s0f2np2
iface enp129s0f2np2 inet manual

auto enp129s0f3np3
iface enp129s0f3np3 inet manual

auto bond0
iface bond0 inet manual
bond-slaves enp129s0f0np0 enp129s0f1np1 enp129s0f2np2 enp129s0f3np3
bond-miimon 100
bond-mode 802.3ad

auto vmbr410
iface vmbr410 inet static
address 10.1.0.211/16
bridge-ports bond0.410
bridge-stp off
bridge-fd 0
#10.1.0.0/16

Bond seems to be established correctly:

Code:

júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: NIC Link is Up, 25000 Mbps full duplex, Flow control: none
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.0 enp129s0f0np0: FEC autoneg off encoding: None
júl 20 13:19:00 sm1 kernel: bond0: (slave enp129s0f0np0): Enslaving as a backup interface with an up link
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: NIC Link is Up, 25000 Mbps full duplex, Flow control: none
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.1 enp129s0f1np1: FEC autoneg off encoding: None
júl 20 13:19:00 sm1 kernel: bond0: (slave enp129s0f1np1): Enslaving as a backup interface with an up link
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.2 enp129s0f2np2: NIC Link is Up, 25000 Mbps full duplex, Flow control: none
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.2 enp129s0f2np2: FEC autoneg off encoding: None
júl 20 13:19:00 sm1 kernel: bond0: (slave enp129s0f2np2): Enslaving as a backup interface with an up link
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.3 enp129s0f3np3: NIC Link is Up, 25000 Mbps full duplex, Flow control: none
júl 20 13:19:00 sm1 kernel: bnxt_en 0000:81:00.3 enp129s0f3np3: FEC autoneg off encoding: None
júl 20 13:19:00 sm1 kernel: bond0: (slave enp129s0f3np3): Enslaving as a backup interface with an up link
júl 20 13:19:00 sm1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready

But only one port is used:

Code:

[sm1@08:36 ~]✝ iperf3 -s -p 9999
-----------------------------------------------------------
Server listening on 9999 (test #1)
-----------------------------------------------------------
Accepted connection from 10.1.0.212, port 53622
[  5] local 10.1.0.211 port 9999 connected to 10.1.0.212 port 53632
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.73 GBytes  23.4 Gbits/sec                 
[  5]   1.00-2.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   2.00-3.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   3.00-4.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   4.00-5.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   5.00-6.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   6.00-7.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   7.00-8.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   8.00-9.00   sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]   9.00-10.00  sec  2.73 GBytes  23.5 Gbits/sec                 
[  5]  10.00-10.00  sec   636 KBytes  22.7 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  27.3 GBytes  23.5 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 9999 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
[sm1@08:39 ~]✝

It might be a switch setup done wrong. But that's my colleague's task.
Any help on setting up the switch or PVE to solve this is appreciated. Thank you very much.

davemcl · Jul 21, 2023

I think you would have to test multiple streams to different hosts due to the hashing algo and even then with the 2nd or 3rd stream theres a 50/50 chance it will just send it down the same link.

aasami · Jul 21, 2023

Using 8 parallel streams gives the same speed unfortunately:

Code:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   239 MBytes  2.01 Gbits/sec    0    721 KBytes      
[  7]   0.00-1.00   sec   248 MBytes  2.08 Gbits/sec    0    721 KBytes      
[  9]   0.00-1.00   sec   224 MBytes  1.88 Gbits/sec    0    638 KBytes      
[ 11]   0.00-1.00   sec   466 MBytes  3.91 Gbits/sec    0   1.19 MBytes      
[ 13]   0.00-1.00   sec   466 MBytes  3.91 Gbits/sec    0   1.19 MBytes      
[ 15]   0.00-1.00   sec   233 MBytes  1.95 Gbits/sec    0    714 KBytes      
[ 17]   0.00-1.00   sec   473 MBytes  3.97 Gbits/sec    0   1.23 MBytes      
[ 19]   0.00-1.00   sec   470 MBytes  3.94 Gbits/sec    0   1.20 MBytes      
[SUM]   0.00-1.00   sec  2.75 GBytes  23.6 Gbits/sec    0

I've tried it from more clients at once with the same result:

Code:

[sm1@09:38 ~]✝ iperf3 -s -1 -D -p4321
[sm1@09:39 ~]✝ iperf3 -s -1 -D -p4322
[sm1@09:39 ~]✝ iperf3 -s -1 -D -p4323

[sm2@09:37 ~]✝ iperf3 -c 10.1.0.211 -p 4321 -i 0 -P 1 -t 30
Connecting to host 10.1.0.211, port 4321
[  5] local 10.1.0.212 port 35044 connected to 10.1.0.211 port 4321
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  28.8 GBytes  8.23 Gbits/sec    0   3.13 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  28.8 GBytes  8.23 Gbits/sec    0             sender
[  5]   0.00-30.00  sec  28.8 GBytes  8.23 Gbits/sec                  receiver

iperf Done.
[sm2@09:40 ~]✝

[sm3@09:22 ~]✝ iperf3 -c 10.1.0.211 -p 4322 -i 0 -P 1 -t 30
Connecting to host 10.1.0.211, port 4322
[  5] local 10.1.0.213 port 40516 connected to 10.1.0.211 port 4322
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  27.9 GBytes  7.97 Gbits/sec    0   3.12 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  27.9 GBytes  7.97 Gbits/sec    0             sender
[  5]   0.00-30.00  sec  27.9 GBytes  7.97 Gbits/sec                  receiver

iperf Done.
[sm3@09:40 ~]✝

[sm4@09:22 ~]✝ iperf3 -c 10.1.0.211 -p 4323 -i 0 -P 1 -t 30
Connecting to host 10.1.0.211, port 4323
[  5] local 10.1.0.214 port 46768 connected to 10.1.0.211 port 4323
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  28.8 GBytes  8.24 Gbits/sec    0   3.39 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  28.8 GBytes  8.24 Gbits/sec    0             sender
[  5]   0.00-30.00  sec  28.8 GBytes  8.24 Gbits/sec                  receiver

iperf Done.
[sm4@09:40 ~]✝

davemcl · Jul 21, 2023

aasami said:

Using 8 parallel streams gives the same speed unfortunately:

Code:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   239 MBytes  2.01 Gbits/sec    0    721 KBytes      
[  7]   0.00-1.00   sec   248 MBytes  2.08 Gbits/sec    0    721 KBytes      
[  9]   0.00-1.00   sec   224 MBytes  1.88 Gbits/sec    0    638 KBytes      
[ 11]   0.00-1.00   sec   466 MBytes  3.91 Gbits/sec    0   1.19 MBytes      
[ 13]   0.00-1.00   sec   466 MBytes  3.91 Gbits/sec    0   1.19 MBytes      
[ 15]   0.00-1.00   sec   233 MBytes  1.95 Gbits/sec    0    714 KBytes      
[ 17]   0.00-1.00   sec   473 MBytes  3.97 Gbits/sec    0   1.23 MBytes      
[ 19]   0.00-1.00   sec   470 MBytes  3.94 Gbits/sec    0   1.20 MBytes      
[SUM]   0.00-1.00   sec  2.75 GBytes  23.6 Gbits/sec    0

No, test streams from multiple hosts - not 1 to 1.

aasami · Jul 21, 2023

I don't think that it's 50/50 chance to use the same link. When one link is fully saturated, it should use another one. Otherwise it wouldn't have any sense to use bond (802.3ad), when there is no gain in performance.

davemcl · Jul 21, 2023

aasami said:
I don't think that it's 50/50 chance to use the same link. When one link is fully saturated, it should use another one. Otherwise it wouldn't have any sense to use bond (802.3ad), when there is no gain in performance.

You should also match the switch hashing algo with your host config.

spirit · Jul 21, 2023

use layer3+4 hash policy (on proxmox node for outgoing traffic, and in your physical swtich for ingoing graffic).

This is the only hash policy able to balance multiple connections between 2 ips. (src ip-dstip-srcport-dstport).

aasami · Jul 21, 2023

Thank's for the tip. Now I've tried layer3+4 hash policy, but the result is unchanged.

Search

Search

Bond setup

aasami

Renowned Member

davemcl

Member

aasami

Renowned Member

davemcl

Member

aasami

Renowned Member

davemcl

Member

spirit

Distinguished Member

aasami

Renowned Member