Slow 10Gb network

R

rbjohnson78

Member
Aug 8, 2022
32
0
11
I have a cluster setup, and doing migrations over the 10 gig bonded network, I'm only seeing an average of 110mb/s during migrations. I have a pool of 5 SSD on zfs for each host, so I'd assuming disk speed is fine. VM's are using virtio of the NIC's. Only thing we have not set is our MTU to 9000, but I wouldn't think that would be limiting us that bad. (Could be wrong)

Any suggestions on the first thing to check?
 
Check duplex and speed. Auto negotiation can be flaky between different NIC vendors and sometimes you need to hard code the settings or the port 'flaps'.
 
MTU won't limit you that bad. I even abandonned playing with that, and stick to the default 1500b because it generated terrible situations, difficult to debug, for close to no benefit.

However, as propose @jamiemoles , definitely check your network interfaces, and make sure the "Errors" counters are steady. Disconnect half of the cable and check. I also confirm the Autonegociation can cause problem. Check with:

Code: 
dmesg -H -w

Next, test with IP or Ethernet benchmark tool to see your effective bandwidth.

I 'll repeat what you red 100x times for your problem. Check the cables. (Thus the idea to use only one at a time).

This is definitely not a Proxmox, but a Linux question.
 
Install iperf3 on all the nodes:

Code: 
apt update
apt install iperf3



On one of the node pve2 in this case:
Code: 
root@pve2:~# iperf3 -s

From another node - pve1 in this case:
Code: 
root@pve1:~# iperf3 -c pve2
Connecting to host pve2, port 5201
[  5] local 192.168.68.5 port 57662 connected to 192.168.68.130 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    376 KBytes     
[  5]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    506 KBytes     
[  5]   2.00-3.00   sec   112 MBytes   935 Mbits/sec    0    506 KBytes     
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec    0    506 KBytes     
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec    0    506 KBytes     
[  5]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    506 KBytes     
[  5]   6.00-7.00   sec   109 MBytes   912 Mbits/sec    0    506 KBytes     
[  5]   7.00-8.00   sec   110 MBytes   925 Mbits/sec    0    506 KBytes     
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    0    506 KBytes     
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec    0    506 KBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

iperf Done.


If OK, check in bidirectionnal:

Code: 
root@pve1:~# iperf3 -c pve2 --bidir
Connecting to host pve2, port 5201
[  5] local 192.168.68.5 port 45566 connected to 192.168.68.130 port 5201
[  7] local 192.168.68.5 port 45574 connected to 192.168.68.130 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   111 MBytes   934 Mbits/sec    0    505 KBytes      
[  7][RX-C]   0.00-1.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   1.00-2.00   sec   111 MBytes   932 Mbits/sec    0    505 KBytes      
[  7][RX-C]   1.00-2.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    505 KBytes      
[  7][RX-C]   2.00-3.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    505 KBytes      
[  7][RX-C]   3.00-4.00   sec   112 MBytes   940 Mbits/sec                
[  5][TX-C]   4.00-5.00   sec   113 MBytes   945 Mbits/sec    0    505 KBytes      
[  7][RX-C]   4.00-5.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   5.00-6.00   sec   112 MBytes   938 Mbits/sec    0    505 KBytes      
[  7][RX-C]   5.00-6.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    505 KBytes      
[  7][RX-C]   6.00-7.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0    505 KBytes      
[  7][RX-C]   7.00-8.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    505 KBytes      
[  7][RX-C]   8.00-9.00   sec   112 MBytes   939 Mbits/sec                
[  5][TX-C]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0    505 KBytes      
[  7][RX-C]   9.00-10.00  sec   112 MBytes   939 Mbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec    0             sender
[  5][TX-C]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0             sender
[  7][RX-C]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

I'm definitely on Gigabit ground obviously ;-)
 
Last edited:
francoisd said:
Install iperf3 on all the nodes:

Code: 
apt update
apt install iperf3



On one of the node pve2 in this case:
Code: 
root@pve2:~# iperf3 -s

From another node - pve1 in this case:
Code: 
root@pve1:~# iperf3 -c pve2
Connecting to host pve2, port 5201
[  5] local 192.168.68.5 port 57662 connected to 192.168.68.130 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    376 KBytes    
[  5]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    506 KBytes    
[  5]   2.00-3.00   sec   112 MBytes   935 Mbits/sec    0    506 KBytes    
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec    0    506 KBytes    
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec    0    506 KBytes    
[  5]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    506 KBytes    
[  5]   6.00-7.00   sec   109 MBytes   912 Mbits/sec    0    506 KBytes    
[  5]   7.00-8.00   sec   110 MBytes   925 Mbits/sec    0    506 KBytes    
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    0    506 KBytes    
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec    0    506 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

iperf Done.


If OK, check in bidirectionnal:

Code: 
root@pve1:~# iperf3 -c pve2 --bidir
Connecting to host pve2, port 5201
[  5] local 192.168.68.5 port 45566 connected to 192.168.68.130 port 5201
[  7] local 192.168.68.5 port 45574 connected to 192.168.68.130 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   111 MBytes   934 Mbits/sec    0    505 KBytes     
[  7][RX-C]   0.00-1.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   1.00-2.00   sec   111 MBytes   932 Mbits/sec    0    505 KBytes     
[  7][RX-C]   1.00-2.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    505 KBytes     
[  7][RX-C]   2.00-3.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0    505 KBytes     
[  7][RX-C]   3.00-4.00   sec   112 MBytes   940 Mbits/sec               
[  5][TX-C]   4.00-5.00   sec   113 MBytes   945 Mbits/sec    0    505 KBytes     
[  7][RX-C]   4.00-5.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   5.00-6.00   sec   112 MBytes   938 Mbits/sec    0    505 KBytes     
[  7][RX-C]   5.00-6.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    505 KBytes     
[  7][RX-C]   6.00-7.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0    505 KBytes     
[  7][RX-C]   7.00-8.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    505 KBytes     
[  7][RX-C]   8.00-9.00   sec   112 MBytes   939 Mbits/sec               
[  5][TX-C]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0    505 KBytes     
[  7][RX-C]   9.00-10.00  sec   112 MBytes   939 Mbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec    0             sender
[  5][TX-C]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0             sender
[  7][RX-C]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

I'm definitely on Gigabit ground obviously ;-)
Click to expand...
Thank you, I will try this.
 
Looks like that is OK
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 1.08 GBytes 9.25 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 0.00-1.00 sec 1.08 GBytes 9.30 Gbits/sec
[ 5][TX-C] 1.00-2.00 sec 1.07 GBytes 9.23 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 1.00-2.00 sec 1.08 GBytes 9.23 Gbits/sec
[ 5][TX-C] 2.00-3.00 sec 1.07 GBytes 9.19 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 2.00-3.00 sec 1.09 GBytes 9.33 Gbits/sec
[ 5][TX-C] 3.00-4.00 sec 1.08 GBytes 9.24 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 3.00-4.00 sec 1.08 GBytes 9.31 Gbits/sec
[ 5][TX-C] 4.00-5.00 sec 1.07 GBytes 9.22 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 4.00-5.00 sec 1.08 GBytes 9.29 Gbits/sec
[ 5][TX-C] 5.00-6.00 sec 1.08 GBytes 9.27 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 5.00-6.00 sec 1.08 GBytes 9.32 Gbits/sec
[ 5][TX-C] 6.00-7.00 sec 1.08 GBytes 9.25 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 6.00-7.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 5][TX-C] 7.00-8.00 sec 1.07 GBytes 9.23 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 7.00-8.00 sec 1.08 GBytes 9.25 Gbits/sec
[ 5][TX-C] 8.00-9.00 sec 1.07 GBytes 9.19 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 8.00-9.00 sec 1.09 GBytes 9.34 Gbits/sec
[ 5][TX-C] 9.00-10.00 sec 1.08 GBytes 9.25 Gbits/sec 0 3.31 MBytes
[ 7][RX-C] 9.00-10.00 sec 1.08 GBytes 9.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-10.00 sec 10.7 GBytes 9.23 Gbits/sec 0 sender
[ 5][TX-C] 0.00-10.00 sec 10.7 GBytes 9.23 Gbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 10.8 GBytes 9.30 Gbits/sec 0 sender
[ 7][RX-C] 0.00-10.00 sec 10.8 GBytes 9.30 Gbits/sec receiver
 
What kind of NIC hardware do you have? We have very strange problems here with Broadcom BCM57504 quad port 10/25 SFP+ NIC:s. Something regarding Linux drivers, I suspect. I don't want to hijack your thread, will open new one today but want nevertheless to know more about your hardware setup.
 
Last edited:
jkarjanl said:
We have very strange problems here with Broadcom BCM57504 quad port 10/25 SFP+ NIC:s.
Click to expand...

Maybe?:

jkarjanl said:
I don't want to hijack your thread, will open new one today [...]
Click to expand...

I would recommend so.
 
Neobin said:
Do you have multiple networks configured on/between the nodes?
If so, did you verify, that the migration traffic actually uses the 10 Gb network?

You can specify the migration network under: "Datacenter" -> "Options" -> "Migration Settings":
https://pve.proxmox.com/pve-docs/datacenter.cfg.5.html -> "migration:"
Click to expand...
We are only using the dual 10 gig links. Not the ideal setup, but these will be reconfigured when our datacenter moves.
 
jkarjanl said:
What kind of NIC hardware do you have? We have very strange problems here with Broadcom BCM57504 quad port 10/25 SFP+ NIC:s. Something regarding Linux drivers, I suspect. I don't want to hijack your thread, will open new one today but want nevertheless to know more about your hardware setup.
Click to expand...
We are using the BCM57412 NetXtreme-E dual 10gb
 
When do a vm migration do same time in shell on both pve hosts this to identify used interface (which could be other as you expected) and throughput :
Maybe first "apt install sysstat" ...
"sar -n DEV 1 --iface=$(ls /sys/devices/pci*/*/*/net/|grep -v pci|xargs|sed s'/ /,/'g)"
 
So, I installed bmon, and even though I have the migration network set to use our 10 gig, it is finding its way to the 1 gig port, which has no IP or is configured for any type of bridge/bond. Very confused.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom