iperf3 speeds 8-10% lower in Proxmox than bare metal?

L

louie1961

Well-Known Member
Jan 25, 2023
434
152
48
I was benchmarking my virtualized TrueNAS setup and I noticed some seemingly unusual behavior in my main proxmox host. When running iperf3 between two different hosts on the same VLAN, but not on the same server, traffic coming out of the main proxmox box is slower than traffic going in:
louie@Admin:~$ iperf3 -c 192.168.50.3
Connecting to host 192.168.50.3, port 5201
[ 5] local 192.168.50.36 port 33222 connected to 192.168.50.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.09 GBytes 9.39 Gbits/sec 11 3.56 MBytes
[ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 3.78 MBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 8 3.98 MBytes
[ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 0 4.00 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.39 Gbits/sec 0 4.00 MBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 0 4.00 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 0 4.00 MBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.39 Gbits/sec 0 4.00 MBytes
[ 5] 8.00-9.00 sec 1.08 GBytes 9.26 Gbits/sec 52 2.93 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec 0 3.20 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 71 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.37 Gbits/sec receiver

iperf Done.
louie@Admin:~$ iperf3 -c 192.168.50.3 -R
Connecting to host 192.168.50.3, port 5201
Reverse mode, remote host 192.168.50.3 is sending
[ 5] local 192.168.50.36 port 42202 connected to 192.168.50.3 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 980 MBytes 8.21 Gbits/sec
[ 5] 1.00-2.00 sec 962 MBytes 8.07 Gbits/sec
[ 5] 2.00-3.00 sec 978 MBytes 8.21 Gbits/sec
[ 5] 3.00-4.00 sec 978 MBytes 8.21 Gbits/sec
[ 5] 4.00-5.00 sec 945 MBytes 7.93 Gbits/sec
[ 5] 5.00-6.00 sec 939 MBytes 7.88 Gbits/sec
[ 5] 6.00-7.00 sec 970 MBytes 8.13 Gbits/sec
[ 5] 7.00-8.00 sec 995 MBytes 8.35 Gbits/sec
[ 5] 8.00-9.00 sec 979 MBytes 8.21 Gbits/sec
[ 5] 9.00-10.00 sec 991 MBytes 8.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.49 GBytes 8.15 Gbits/sec 19290 sender
[ 5] 0.00-10.00 sec 9.49 GBytes 8.15 Gbits/sec receiver
Click to expand...

I tried two different NICs with the same effect (both Intel X520-DA2 clones from Amazon). My set up is that my main proxmox host is connected by two AOC connections using LACP to my switch. The other host is connected via a 10 base T connection (no LAG, one connection only). It seems like the outbound traffic from my main node is generating a lot of retries.

Is this just a peculiarity of the Intel X520 cards, or is there a configuration I could tweak to address the retries and get the inbound and outbound traffic up to approximately the same speed? If I run iperf3 -c 192.168.50.XX -P 5, I get 9.35 Gbits/second both ways. Most of my network traffic is data moving over NFS. Does NFS use multithreading? Also I don't see this type of behavior on my other other Proxmox node which has a single 10 base T NIC in it (acquantia chipset I believe)
 
Here's a bit more information. When I run iperf3 from the terminal of my proxmox hosts, I get great network performance

going fom pve01 to pve02.jpg

going fom pve02 to pve01.jpg

However, when I run iperf3 from a VM on host one to a vm on host two, the results are not nearly as good

going from vm on pve-02 to vm on pve-01.jpg

going from vm on pve-01 to vm on pve-02.jpg

My hosts are both on the same management VLAN, and the VMs I used are also on the same VLAN as each other (although a different VLAN from the management VLAN. Host one has the Intel X520 NIC and host 2 has an acquantia 10baseT NIC. To me this clearly points to the difference arising from some aspect of Proxmox. And it affects the outbound traffic of the X520 NIC more. I mean it seems I am losing 3% of the packets in that direction. Is there some configuration in Proxmox I could check/adjust to address this?
 
Hi louie1961,

could you check the results of iperf3 against those of iperf, can you see a difference there?
 
In iperf I am getting 9.37 and 9.39 gbits/sec from host to host (and in reverse). From VM to VM (in the same VLAN) I am seeing 8.83 and 9.33. So about 6% difference
 
Hi louie1961,

correct me if I'm wrong:
+ you have two PVE hosts, one connected to your switch using an LACP bond (PVE-host1), the other one (PVE-host2) using a single connection is attached to the same switch.
+ iperf tests between PVE-host1<->PVE-host2 show ~9.4Gbit/s
+ iperf tests between VM01 on PVE-host1<->VM02 on PVE-host2 show ~9.3Gbit/s, respectively ~8.8Gbit/s using the same bridge as the tests for the hosts.

Are these VMs set up identically? From which side Is the 'lower' bandwidth (8.8Gbit/s) measured from -- is this `iperf3 -c [...]` triggered from VM01 (on PVE-host1) or VM02 (on PVE-host2)?

Is there any other traffic in your 192.168.50.3/24(?) network that might interfere?

Is the behaviour the same if you use the same NIC-setup on both of your involved PVE-hosts (if applicable)? Are you using the same MTU settings in every place?

Is your switch logging anything of interest, when you're doing the tests?

Clearly there is some space for improvement -- but generally I wouldn't call your numbers exactly bad.
 
Hi Daniel I appreciate your response. You are correct that my numbers are not that bad. But I had it in my head that I would lose only 1-3% performance being virtualized and I only see this on the one host using the intel X520 card. I can live with the performance if need be, but my intellectual curiosity is aroused at this point, so if I can fix the problem, that would be pretty satisfying.

you have two PVE hosts, one connected to your switch using an LACP bond (PVE-host1), the other one (PVE-host2) using a single connection is attached to the same switch.
+ iperf tests between PVE-host1<->PVE-host2 show ~9.4Gbit/s
+ iperf tests between VM01 on PVE-host1<->VM02 on PVE-host2 show ~9.3Gbit/s, respectively ~8.8Gbit/s using the same bridge as the tests for the hosts.
Click to expand...
All correct. Host #1 has a Intel X520 card and Host #2 has an acquantia NIC. The slowdown seems to be on the outbound side of Host #1 (e.g. on the outbound side of the X520 NIC)
Are these VMs set up identically? From which side Is the 'lower' bandwidth (8.8Gbit/s) measured from -- is this `iperf3 -c [...]` triggered from VM01 (on PVE-host1) or VM02 (on PVE-host2)?
Click to expand...
Yes. All running Ubuntu server, and I equalized the cores and memory between the sending VM and the receiving VM. It seems to happen regardless of which VM I choose to use on host #1. I have tried several different ones. I also specifically spun up new Debian 12 VMs to act as sender and receiver on each host, with the same results.
Is your switch logging anything of interest, when you're doing the tests?
Click to expand...

I will have to check and see

Is there any other traffic in your 192.168.50.3/24(?) network that might interfere?
Click to expand...

Its always a possibility that I missed something, but when I spun up the Debian test VMs I shut all the other ones down

Is the behaviour the same if you use the same NIC-setup on both of your involved PVE-hosts (if applicable)? Are you using the same MTU settings in every place?
Click to expand...
I am unable to use the same NIC setup on both sides. Host #1 is a PC built on a mini ITX motherboard. I have tried it with two different X520 based NICs and see the same behavior. Host #2 is an HP Elitedesk Mini PC. It has no PCI slot. MTU is set to 1500 everywhere (switch, router, proxmox hosts)
 
You must log in or register to reply here.
Top Bottom