10 gbe Transfer speed asymetrical

[groenni]

Hi,
Testing network throughput with iperf3, I notice receive is a lot slower than transmit.
Tried some optimizations but can not fix this.
Running latest version of ProxMox, NIC is a Chelsio T3.
Same results back to back or over a switch. Replaced the cable as well, Optical or DAC, same results.
No firewall rules yet.

This is in the ProxMox pve host shell:


root@pve:~# iperf3 -c 192.168.100.111 -M 9000 -i 10 -t 10 -l 1m -w 4m
Connecting to host 192.168.100.111, port 5201
[ 5] local 192.168.100.101 port 57258 connected to 192.168.100.111 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 1223 4.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 1223 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver

iperf Done.
root@pve:~# iperf3 -c 192.168.100.111 -M 9000 -i 10 -t 10 -l 1m -w 4m -R
Connecting to host 192.168.100.111, port 5201
Reverse mode, remote host 192.168.100.111 is sending
[ 5] local 192.168.100.101 port 57262 connected to 192.168.100.111 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.16 GBytes 5.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.16 GBytes 5.29 Gbits/sec sender
[ 5] 0.00-10.00 sec 6.16 GBytes 5.29 Gbits/sec receiver

iperf Done.

 

[Bidi]

Hello,

For us is even strange we buyet 10GB SFP+ switch + Network Cards added on 2 servers ( 1. PowerEdge R630 Brand New server with SSDs, DDR4...etc 2. PowerEdge R810 4 x 10 Core E7-4870 ...etc) and when we migrate VMs From one node to other it takes 5 Minutes :| for am 20GB Vm. and the speed betwin them is only 2GB and we dont uderstand why.


2021-01-06 11:38:25 starting migration of VM 167 to node 'd7' (............)
2021-01-06 11:38:25 found generated disk 'local:167/vm-167-cloudinit.qcow2' (in current VM config)
2021-01-06 11:38:25 found local disk 'localssd:167/vm-167-disk-0.raw' (in current VM config)
2021-01-06 11:38:25 copying local disk images
2021-01-06 11:38:27 Formatting '/home2/images/167/vm-167-disk-0.raw', fmt=raw size=21474836480
2021-01-06 11:40:11 5242880+0 records in
2021-01-06 11:40:11 5242880+0 records out
2021-01-06 11:40:11 21474836480 bytes (21 GB, 20 GiB) copied, 103.465 s, 208 MB/s
2021-01-06 11:43:47 6764+1247337 records in
2021-01-06 11:43:47 6764+1247337 records out
2021-01-06 11:43:47 21474836480 bytes (21 GB, 20 GiB) copied, 319.695 s, 67.2 MB/s
2021-01-06 11:43:47 successfully imported 'localssd:167/vm-167-disk-0.raw'
2021-01-06 11:43:47 volume 'localssd:167/vm-167-disk-0.raw' is 'localssd:167/vm-167-disk-0.raw' on the target
2021-01-06 11:43:51 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=d7' root@............... pvesr set-state 167 \''{}'\'
2021-01-06 11:43:54 migration finished successfully (duration 00:05:30)
TASK OK


All servers ar SSDs with RAID PERC H730 & H710 , HP NC552SFP ETHERNET 10GB 2-PORT SFP+

All cables ar Mikrotik DAC and the Switch two an CRS354-48G-4S+2Q+RM