2x40Gb bond speed only at 40Gb

[petruzzo]

Hi I have configure a 6 node cluster with this packages

Spoiler: pveversion --verbose

proxmox-ve: 7.2-1 (running kernel: 5.15.60-2-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-helper: 7.2-13
pve-kernel-5.15: 7.2-12
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 17.2.4-pve1
ceph-fuse: 17.2.4-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-3
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-4
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

on each node I have a dual port mellanox connectx3 card

Spoiler: lshw --class network

.......
*-network
description: Ethernet interface
product: MT27500 Family [ConnectX-3]
vendor: Mellanox Technologies
physical id: 0
bus info: pci@0000:5d:00.0
logical name: eno5
version: 00
serial: 24:be:05:a6:a7:c1
capacity: 56Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx4_en driverversion=4.0-0 duplex=full firmware=2.42.5030 latency=0 link=yes multicast=yes
resources: irq:34 memory:ea000000-ea0fffff memory:e8000000-e9ffffff memory:cffe0000000-cffffffffff
*-network:0
description: Ethernet interface
physical id: 3
bus info: pci@0000:5d:00.0
logical name: eno5d1
serial: 24:be:05:a6:a7:c2
capacity: 56Gbit/s
capabilities: ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx4_en driverversion=4.0-0 duplex=full firmware=2.42.5030 link=yes multicast=yes
......

the ports are connected to a 40Gb switch via DAC cable and LACP configured

on each server I configure bond with ovs

Spoiler: cat /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto eno5
iface eno5 inet manual
mtu 9000

auto eno5d1
iface eno5d1 inet manual
mtu 9000

auto vlan10
iface vlan10 inet static
address ....
gateway ....
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options tag=10

auto vlan9
iface vlan9 inet static
address ....
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options tag=9

auto van12
iface van12 inet static
address .....
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options tag=12

auto vlan4
iface vlan4 inet static
address .....
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options tag=4

auto bond0
iface bond0 inet manual
ovs_bonds eno5 eno5d1
ovs_type OVSBond
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options lacp=active bond_mode=balance-slb

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 vlan10 vlan9 van12 vlan4
ovs_mtu 9000

I tested the link between the servers with iperf3 with 3 parallel process as this guide and with this resoults

Spoiler: iperf

s3: - - - - - - - - - - - - - - - - - - - - - - - - -
s3: [ ID] Interval Transfer Bitrate Retr
s3: [ 5] 0.00-10.00 sec 21.1 GBytes 18.1 Gbits/sec 29 sender
s3: [ 5] 0.00-10.04 sec 21.1 GBytes 18.1 Gbits/sec receiver
s3:
s3: iperf Done.
s1: [ 5] 9.00-10.00 sec 1.32 GBytes 11.4 Gbits/sec 0 1.53 MBytes
s1: - - - - - - - - - - - - - - - - - - - - - - - - -
s1: [ ID] Interval Transfer Bitrate Retr
s1: [ 5] 0.00-10.00 sec 12.4 GBytes 10.7 Gbits/sec 8 sender
s1: [ 5] 0.00-10.04 sec 12.4 GBytes 10.6 Gbits/sec receiver
s1:
s1: iperf Done.
s2: [ 5] 9.00-10.00 sec 1.36 GBytes 11.7 Gbits/sec 1 1.48 MBytes
s2: - - - - - - - - - - - - - - - - - - - - - - - - -
s2: [ ID] Interval Transfer Bitrate Retr
s2: [ 5] 0.00-10.00 sec 12.3 GBytes 10.6 Gbits/sec 23 sender
s2: [ 5] 0.00-10.04 sec 12.3 GBytes 10.5 Gbits/sec receiver
s2:
s2: iperf Done.

I try to set the governor of cpu but I cant set it because cpupower tell me that no governor as avaible

Spoiler: cpupower frequency-info

analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
boost state support:
Supported: yes
Active: yes

Someone can help me

 

[vesalius]

https://forum.proxmox.com/threads/lacp-bond-wihout-speed-increase.54955/

 

[spirit]

@petruzzo

be carefull than iperf3 is not multithreaded, so it's possible that you have 1core at 100% cpu.

you need to use iperf2 to use multithreaded process with parallel connection, or launch multiple iperf3 command in //.

also, you should use "balance-tcp" instead "balance-slb" to load balance multiple connections for outbound traiffic on multiples links between 2 servers. (and same on your physical switch to loadbalance the inbound traffic).

 

[petruzzo]

hi
I launch multiple iperf3 server and client to test
after change from balance-slb to balance-tcp the thougput now perform only 46Gbs

 

[spirit]

after change from balance-slb to balance-tcp the thougput now perform only 46Gbs

do you have done the same on your swith ? (this can be called hash-policy, loadbalance-policy, with somethime ipsrc-portsec-ipdest-portdest).
It's really depend on the switch brand && model. (and some lowcost swtich don't support balancing of ip-port , only ip-mac).

This is if you want to test between 2 servers/ip.


Now, with balance-slb, you should be able to launch multiple iperf on 1 server,with different ips/servers for destination, it should loadbalance between 2 links.