Bond Performance

vitor costa

Active Member
Oct 28, 2009
142
2
38
I trying install a cluster with two nodes. Each node with 3 network cards. One card to service client network and 2 others to create a bond to drbd sincronizations and cluster comiunication. Bond work but no performance gain observed. I used iperf to test speed on bond link and value is 1 gb/s like only one card in operation. I tryed all bond protocols including switch assisted. Switch show lacp working as expected.
 

Raymond Burns

Member
Apr 2, 2013
333
1
18
Houston, Texas, United States
I recently had this issue.
To see a performance increase, my understanding, is to use 802.3ad, then you MUST set the ports used on your switch to LACP and group or trunk them together.
The other options are for failover options which you should see if you disconnect one port.
For 802.3ad you must set the switch to active-active LACP and trunk the ports together.
What switch is it? I can attest that it is working properly, as I am receiving 1.7mbit throughput from all hosts to NAS.
 

spirit

Famous Member
Apr 2, 2010
5,686
626
133
www.odiso.com
I trying install a cluster with two nodes. Each node with 3 network cards. One card to service client network and 2 others to create a bond to drbd sincronizations and cluster comiunication. Bond work but no performance gain observed. I used iperf to test speed on bond link and value is 1 gb/s like only one card in operation. I tryed all bond protocols including switch assisted. Switch show lacp working as expected.

Hi, with lacp you can't do more than 1gbit/s by tcp connection.

multiple tcp connections will be balanced between 2 links. (but 1tcp connection is never balanced to reach 2gbit)
 

Raymond Burns

Member
Apr 2, 2013
333
1
18
Houston, Texas, United States
That can't be true, unless I do not connect to my NAS through TCP. But I can verify 1.8gbit transmission from Proxmox to OmniOS/Napp-IT during a "Move Disk" for a VM.
If I do not connect through TCP, which I don't think it does because my type=NFSv3 and not =NFSv3,tcp; is iPERF a TCP benchmark only?
 

vitor costa

Active Member
Oct 28, 2009
142
2
38
I recently had this issue.
To see a performance increase, my understanding, is to use 802.3ad, then you MUST set the ports used on your switch to LACP and group or trunk them together.
The other options are for failover options which you should see if you disconnect one port.
For 802.3ad you must set the switch to active-active LACP and trunk the ports together.
What switch is it? I can attest that it is working properly, as I am receiving 1.7mbit throughput from all hosts to NAS.

Its a TL-SG3216 . I attached lacp screen

switch-tplink.png
 

vitor costa

Active Member
Oct 28, 2009
142
2
38
In this case DRBD with lacp is no way ? In a site a guy sai best use balance-rr protocol w crossover cable (no switch). What u thinj ?
 

Raymond Burns

Member
Apr 2, 2013
333
1
18
Houston, Texas, United States
...WAIT...
I am quite sure this is true (for LACP). But maybe you storage layer opens more than one tcp connection.
This is what I did:
Set a bond in Napp-IT using eth0 and eth1 going to port 1 and 2 on my DGS-3200.
Set a bond in Proxmox using 802.3ad for eth1 and eth2 going to port 3 and 4 on my DGS-3200.
I set my DGS-3200 to have Active-Active LACP for ports 1-2-3-4
Set DGS-3200 to Port Trunk as follows:
Trunk 1 = Port 1-2
Trunk 2 = Port 3-4

When I move a full VM disk from local storage to NFS I see 1.7-1.8gbit transmission in "top" and in the Napp-IT GUI under the monitor. I do not see this reflected in the Proxmox GUI, but I assume the GUI is not calculating my bond, but my management port which is eth0.

I have not done any additional setting to my OmniOS as in change the type to NFSv3,tcp.
Please explain what is the difference from what the OP is trying to do and/or how I'm not calculating the transmission speed properly.
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
17,124
521
133
Austria
www.proxmox.com
Please explain what is the difference from what the OP is trying to do and/or how I'm not calculating the transmission speed properly.

AFAIK DRBD use a single tcp connection (at least his results suggests that).
 

vitor costa

Active Member
Oct 28, 2009
142
2
38
iperf can do multiple tcp connections with -P parameter. I try using it and each connection show half speed, total keep 1 gb/s. So this multiple tcp connections dont work too.
 

vitor costa

Active Member
Oct 28, 2009
142
2
38
i believe -d do a bidirecional test... Result is 1 gb/s for each side... w -d -P 2 i receive again a SUM with 1 gb/s
 

spirit

Famous Member
Apr 2, 2010
5,686
626
133
www.odiso.com

jbennet

Member
Jul 19, 2013
18
0
21
France
If that can be of any interest, I post my lacp bonding configuration over 4 ethernet cards.
The switch MUST also be configured as LACP on theses ports.
With layer 3/4 xmit hash policy it might interestingly balance multiple tcp connections between the same hosts.

Code:
auto lo
iface lo inet loopback

iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual

auto bond0
iface bond0 inet manual
    slaves eth0 eth1 eth2 eth3
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
    address 172.16.20.10
    netmask 255.255.0.0
    gateway 172.16.50.1
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
    post-up for i in eth0 eth1 eth2 eth3 ; do for o in rx ; do ethtool -K $i $o off; done; done
 

jbennet

Member
Jul 19, 2013
18
0
21
France
That is normal behaviour. LACP is designed to _balance_ bandwidth between the links, but the maximum throughput between two clients is 1Gb/s.

(What is a "client" depend on the method used to compute which link should be used in function of the layer 2/3/4 adresses ;
e.g. with layer 2 you always use the same link between two ethernet cards).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!