iperf3 on 10G lots of Retr

[flexyz]

Hi

I am running proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve) on two hosts and have connected them together directly

And when I test speeds with iperf3, I see a lot of "retr" values?

pve1:
auto enp94s0f1
iface enp94s0f1 inet static
address 10.15.15.12
netmask 255.255.255.0

pve2:
auto enp94s0f1
iface enp94s0f1 inet static
address 10.15.15.13
netmask 255.255.255.0


from pve1 to pve2:

# iperf3 -c 10.15.15.13
Connecting to host 10.15.15.13, port 5201
[ 5] local 10.15.15.12 port 57916 connected to 10.15.15.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 281 1.62 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.62 MBytes
[ 5] 2.00-3.00 sec 1.08 GBytes 9.29 Gbits/sec 901 1.22 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.52 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.34 Gbits/sec 67 1.60 MBytes
[ 5] 5.00-6.00 sec 1.08 GBytes 9.30 Gbits/sec 745 1.24 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.49 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.52 MBytes
[ 5] 8.00-9.00 sec 1.08 GBytes 9.23 Gbits/sec 984 1.40 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec 2978 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec receiver


from pve2 to pve1:

# iperf3 -c 10.15.15.12
Connecting to host 10.15.15.12, port 5201
[ 5] local 10.15.15.13 port 55058 connected to 10.15.15.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.71 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.71 MBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.35 Gbits/sec 163 1.85 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 66 1.39 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.36 Gbits/sec 47 1.51 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 276 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver

 

[mac.linux.free]

I had similar issues...give OpenVSwitch a try.

 

[flexyz]

this is without any bridge,vlan etc. just direct connected

I will look at the OpenVSwitch, but is not preferred

Felix

 

[mac.linux.free]

works at last for me

 

[flexyz]

Any feedback from Proxmox support?

 

[flexyz]

Please proxmox support - I just want to know if there is something I might could try before I change to OpenVSwitch

Felix

 

[Dunuin]

You could limit the iperf3 speed to half of that to see if it still occures. I've only seen packetloss here when NIC/CPU are at the limits and can't keep up with all the packets.

 

[flexyz]

Thanks, here are the results - I have fast CPUs and lots of memory, so something else must be wrong

From the same VM I get 28-306GB without any retries

 

[spirit]

@flexyz

maybe can you try to increase linux buffer ?

something like with sysctl.conf

# allow testing with buffers up to 64MB 
net.core.rmem_max = 67108864 
net.core.wmem_max = 67108864 
# increase Linux autotuning TCP buffer limit to 32MB
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

also enable bbr congestion control

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

 

[flexyz]

Here are the results, still not good. First dump is between pve1<->pve2, second is from vm-pve1<->vm-pve2

First run on pve1<->pve2 was without any retries, following tests with

Thanks again
Felix