iperf3 on 10G lots of Retr

flexyz

Well-Known Member
Sep 22, 2016
154
9
58
53
Hi

I am running proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve) on two hosts and have connected them together directly

And when I test speeds with iperf3, I see a lot of "retr" values?

pve1:
auto enp94s0f1
iface enp94s0f1 inet static
address 10.15.15.12
netmask 255.255.255.0

pve2:
auto enp94s0f1
iface enp94s0f1 inet static
address 10.15.15.13
netmask 255.255.255.0


from pve1 to pve2:

# iperf3 -c 10.15.15.13
Connecting to host 10.15.15.13, port 5201
[ 5] local 10.15.15.12 port 57916 connected to 10.15.15.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 281 1.62 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.62 MBytes
[ 5] 2.00-3.00 sec 1.08 GBytes 9.29 Gbits/sec 901 1.22 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.52 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.34 Gbits/sec 67 1.60 MBytes
[ 5] 5.00-6.00 sec 1.08 GBytes 9.30 Gbits/sec 745 1.24 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.49 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.52 MBytes
[ 5] 8.00-9.00 sec 1.08 GBytes 9.23 Gbits/sec 984 1.40 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec 2978 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec receiver


from pve2 to pve1:

# iperf3 -c 10.15.15.12
Connecting to host 10.15.15.12, port 5201
[ 5] local 10.15.15.13 port 55058 connected to 10.15.15.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.35 Gbits/sec 163 1.85 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 66 1.39 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.36 Gbits/sec 47 1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 276 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver
 
You could limit the iperf3 speed to half of that to see if it still occures. I've only seen packetloss here when NIC/CPU are at the limits and can't keep up with all the packets.
 
Last edited:
Thanks, here are the results - I have fast CPUs and lots of memory, so something else must be wrong

From the same VM I get 28-306GB without any retries
 

Attachments

  • Screenshot 2021-12-07 at 08.12.45.png
    Screenshot 2021-12-07 at 08.12.45.png
    196.7 KB · Views: 51
  • Screenshot 2021-12-07 at 08.14.44.png
    Screenshot 2021-12-07 at 08.14.44.png
    70.2 KB · Views: 46
@flexyz

maybe can you try to increase linux buffer ?

something like with sysctl.conf

Code:
# allow testing with buffers up to 64MB 
net.core.rmem_max = 67108864 
net.core.wmem_max = 67108864 
# increase Linux autotuning TCP buffer limit to 32MB
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

also enable bbr congestion control

Code:
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
 
Here are the results, still not good. First dump is between pve1<->pve2, second is from vm-pve1<->vm-pve2

First run on pve1<->pve2 was without any retries, following tests with

Thanks again
Felix
 

Attachments

  • Screenshot 2021-12-07 at 09.34.42.png
    Screenshot 2021-12-07 at 09.34.42.png
    195.3 KB · Views: 62
  • Screenshot 2021-12-07 at 09.35.00.png
    Screenshot 2021-12-07 at 09.35.00.png
    137.3 KB · Views: 62

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!