iperf3 retries on VMs

[barryyancey]

I am running a PVE cluster (3) on 6.8.12-4. I have a Dell R730XD, and 2 R630s. 2 boxes have the Intel(R) 10GbE 4P X710 rNDC, and the other has the BRCM 10G/GbE 2+2P 57800 rNDC. iperf3 runs great between my NAS and PVE hosts (and their containers), but not on the VMs runnings on those hosts. The throughput is good, but as you can see, the retries are through the roof!

Any ideas???

Performance between the NAS (10.0.1.100) and a PVE host (apollo - 10.0.1.102) is great.



Performance between the PVE host (apollo - 10.0.1.102) and a VM (prod-plex - 10.0.1.105) is great. That VM is on the apollo PVE host.



Performance between a VM (prod-plex - 10.0.1.105) and the NAS (10.0.1.100) is poor. The NAS is a Dell R720XD with an Intel(R) 10GbE 4P X710 rNDC.



This is obviously not a switching issue as the PVE hosts and my NAS are communicating perfectly.

Please see the usage on the PVE hosts.

 

[barryyancey]

Any ideas???

 

[jomm81]

I'm having the same issue, were you able to find a solution?

 

[barryyancey]

Unfortunately, no.

 

[jpeppard]

I've also just started noticing this.

Iperf3 has very high retries when transiting between VMs in different VLANs, meaning traffic leaving a VM in VLAN1 out to the router and back into a VM in VLAN2 has very high retry. All VMs are debian based and using Virtio drivers.

Iperf3 between VMs in the same VLAN on the same proxmox host have 0 retries, great speed.

I am using a Mellanox X3 card with SFP+ Twinax 10G cabling.

I can't tell if this was always the case or just came up in recent kernels.


Going to upgrade to a Mellanox X4, upgrade the firmware and see what happens.

 

[john2069]

I found the same issue between my VMs and another host, however I'm only seeing high retries when SENDING from the VM. Receiving to the VM from another host runs perfectly fine.

 

[john2069]

@barryyancey - are you able to run your same test but add the -R option to the iperf command to switch sender/receiver?

 

[gregrob]

@john2069 @barryyancey - I've observed similar behaviour with my Debian VMs - specifically when initiating traffic from the VM to another host. Interestingly, running iperf3 in reverse mode (-R) shows no issue, which suggests the bottleneck is on the transmit path from the VM.

What helped significantly was tuning some TCP buffer and congestion control settings inside the VM itself. I'm not entirely sure if this is a true fix, as I don't fully understand the root cause of the issue - but it has definitely improved performance in my case:

sudo sysctl -w net.core.rmem_max=12582912
sudo sysctl -w net.core.wmem_max=12582912
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 12582912"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 12582912"
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.ipv4.tcp_mtu_probing=1

If this works for you too, you could persist the changes in a .conf file under /etc/sysctl.d/

 

[jpeppard]

To close out my issue reported above, it ended up being my Mikrotik router which has insufficient packet buffering at the interface hardware level. I implemented flow control between the router and switch which has helped significantly with packet drops.