Wildly different TCP vs UDP iperf3 results between guests on same hypervisor

B

boethius

Renowned Member
Feb 11, 2016
36
4
73
41
Hello,

I am trying to understand my iperf3 results for my entire network, but even between guests on same Proxmox node it is not making sense to me...

10.44.35.128 is a Ubuntu 20.04 Desktop VM
10.44.35.102 is a Ubuntu 20.04 Server VM

I am seeing UDP results sometimes as low as 400Mbps with 50-80% packet loss. Sometimes as high as slightly over 1Gbps with <1% packet loss.
TCP rates at times as high as 20Gbps for an interval of 1s, but the result of tests are around 13-15Gbps for the entire 10s.

I don't know what to make of it. How is there packet loss if these packets are not leaving the host? Is there something I am doing wrong in terms of my iperf commands?



Code: 
Server listening on 5201

-----------------------------------------------------------

Accepted connection from 10.44.35.102, port 40604

[  5] local 10.44.35.128 port 5201 connected to 10.44.35.102 port 40606

[  8] local 10.44.35.128 port 5201 connected to 10.44.35.102 port 40608

[ 10] local 10.44.35.128 port 5201 connected to 10.44.35.102 port 40610

[ 12] local 10.44.35.128 port 5201 connected to 10.44.35.102 port 40612

[ ID] Interval           Transfer     Bitrate

[  5]   0.00-1.00   sec   471 MBytes  3.95 Gbits/sec              

[  8]   0.00-1.00   sec   488 MBytes  4.09 Gbits/sec              

[ 10]   0.00-1.00   sec   468 MBytes  3.93 Gbits/sec              

[ 12]   0.00-1.00   sec   471 MBytes  3.95 Gbits/sec              

[SUM]   0.00-1.00   sec  1.85 GBytes  15.9 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   1.00-2.00   sec   401 MBytes  3.36 Gbits/sec              

[  8]   1.00-2.00   sec   401 MBytes  3.36 Gbits/sec              

[ 10]   1.00-2.00   sec   401 MBytes  3.36 Gbits/sec              

[ 12]   1.00-2.00   sec   401 MBytes  3.36 Gbits/sec              

[SUM]   1.00-2.00   sec  1.56 GBytes  13.4 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   2.00-3.00   sec   475 MBytes  3.99 Gbits/sec              

[  8]   2.00-3.00   sec   476 MBytes  3.99 Gbits/sec              

[ 10]   2.00-3.00   sec   475 MBytes  3.99 Gbits/sec              

[ 12]   2.00-3.00   sec   474 MBytes  3.98 Gbits/sec              

[SUM]   2.00-3.00   sec  1.86 GBytes  15.9 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   3.00-4.00   sec   490 MBytes  4.11 Gbits/sec              

[  8]   3.00-4.00   sec   489 MBytes  4.10 Gbits/sec              

[ 10]   3.00-4.00   sec   490 MBytes  4.11 Gbits/sec              

[ 12]   3.00-4.00   sec   488 MBytes  4.09 Gbits/sec              

[SUM]   3.00-4.00   sec  1.91 GBytes  16.4 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   4.00-5.00   sec   479 MBytes  4.01 Gbits/sec              

[  8]   4.00-5.00   sec   480 MBytes  4.02 Gbits/sec              

[ 10]   4.00-5.00   sec   478 MBytes  4.01 Gbits/sec              

[ 12]   4.00-5.00   sec   479 MBytes  4.02 Gbits/sec              

[SUM]   4.00-5.00   sec  1.87 GBytes  16.1 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   5.00-6.00   sec   482 MBytes  4.04 Gbits/sec              

[  8]   5.00-6.00   sec   482 MBytes  4.05 Gbits/sec              

[ 10]   5.00-6.00   sec   481 MBytes  4.03 Gbits/sec              

[ 12]   5.00-6.00   sec   480 MBytes  4.02 Gbits/sec              

[SUM]   5.00-6.00   sec  1.88 GBytes  16.1 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   6.00-7.00   sec   514 MBytes  4.31 Gbits/sec              

[  8]   6.00-7.00   sec   513 MBytes  4.30 Gbits/sec              

[ 10]   6.00-7.00   sec   513 MBytes  4.30 Gbits/sec              

[ 12]   6.00-7.00   sec   514 MBytes  4.31 Gbits/sec              

[SUM]   6.00-7.00   sec  2.01 GBytes  17.2 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   7.00-8.00   sec   410 MBytes  3.44 Gbits/sec              

[  8]   7.00-8.00   sec   411 MBytes  3.44 Gbits/sec              

[ 10]   7.00-8.00   sec   410 MBytes  3.43 Gbits/sec              

[ 12]   7.00-8.00   sec   409 MBytes  3.43 Gbits/sec              

[SUM]   7.00-8.00   sec  1.60 GBytes  13.7 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   8.00-9.00   sec   439 MBytes  3.69 Gbits/sec              

[  8]   8.00-9.00   sec   438 MBytes  3.68 Gbits/sec              

[ 10]   8.00-9.00   sec   439 MBytes  3.69 Gbits/sec              

[ 12]   8.00-9.00   sec   439 MBytes  3.68 Gbits/sec              

[SUM]   8.00-9.00   sec  1.71 GBytes  14.7 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[  5]   9.00-10.00  sec   390 MBytes  3.27 Gbits/sec              

[  8]   9.00-10.00  sec   390 MBytes  3.27 Gbits/sec              

[ 10]   9.00-10.00  sec   390 MBytes  3.27 Gbits/sec              

[ 12]   9.00-10.00  sec   390 MBytes  3.27 Gbits/sec              

[SUM]   9.00-10.00  sec  1.52 GBytes  13.1 Gbits/sec              

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate

[  5]   0.00-10.00  sec  4.44 GBytes  3.82 Gbits/sec                  receiver

[  8]   0.00-10.00  sec  4.46 GBytes  3.83 Gbits/sec                  receiver

[ 10]   0.00-10.00  sec  4.44 GBytes  3.81 Gbits/sec                  receiver

[ 12]   0.00-10.00  sec  4.44 GBytes  3.81 Gbits/sec                  receiver

[SUM]   0.00-10.00  sec  17.8 GBytes  15.3 Gbits/sec                  receiver

-----------------------------------------------------------





However, my UDP results are ...





Code: 
joe@ubuntuserver:~$ iperf3 -u -b 0 -Z -c 10.44.35.128

Connecting to host 10.44.35.128, port 5201

[  5] local 10.44.35.102 port 39515 connected to 10.44.35.128 port 5201

[ ID] Interval           Transfer     Bitrate         Total Datagrams

[  5]   0.00-1.00   sec   117 MBytes   985 Mbits/sec  85050

[  5]   1.00-2.00   sec   120 MBytes  1.01 Gbits/sec  86990

[  5]   2.00-3.00   sec   117 MBytes   980 Mbits/sec  84600

[  5]   3.00-4.00   sec   122 MBytes  1.02 Gbits/sec  88040

[  5]   4.00-5.00   sec   113 MBytes   948 Mbits/sec  81870

[  5]   5.00-6.00   sec   107 MBytes   899 Mbits/sec  77560

[  5]   6.00-7.00   sec   113 MBytes   948 Mbits/sec  81850

[  5]   7.00-8.00   sec   116 MBytes   972 Mbits/sec  83910

[  5]   8.00-9.00   sec   119 MBytes   998 Mbits/sec  86130

[  5]   9.00-10.00  sec   121 MBytes  1.02 Gbits/sec  87930

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datag                                                        rams

[  5]   0.00-10.00  sec  1.14 GBytes   978 Mbits/sec  0.000 ms  0/843930 (0%)  s                                                        ender

[  5]   0.00-10.00  sec   865 MBytes   726 Mbits/sec  0.012 ms  217497/843840 (2                                                        6%)  receiver



iperf Done.

joe@ubuntuserver:~$ iperf3 -u -b 0 -R -Z -c 10.44.35.128

Connecting to host 10.44.35.128, port 5201

Reverse mode, remote host 10.44.35.128 is sending

[  5] local 10.44.35.102 port 54549 connected to 10.44.35.128 port 5201

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-1.00   sec  98.9 MBytes   830 Mbits/sec  0.006 ms  3134/74770 (4.2%)

[  5]   1.00-2.00   sec  86.3 MBytes   724 Mbits/sec  0.006 ms  685/63212 (1.1%)

[  5]   2.00-3.00   sec  85.0 MBytes   713 Mbits/sec  0.010 ms  675/62218 (1.1%)

[  5]   3.00-4.00   sec  93.5 MBytes   784 Mbits/sec  0.012 ms  1040/68761 (1.5%)

[  5]   4.00-5.00   sec  93.0 MBytes   780 Mbits/sec  0.009 ms  1414/68780 (2.1%)

[  5]   5.00-6.00   sec  89.5 MBytes   751 Mbits/sec  0.010 ms  154/64945 (0.24%)

[  5]   6.00-7.00   sec  95.1 MBytes   797 Mbits/sec  0.008 ms  917/69750 (1.3%)

[  5]   7.00-8.00   sec  91.8 MBytes   770 Mbits/sec  0.010 ms  918/67421 (1.4%)

[  5]   8.00-9.00   sec  87.1 MBytes   731 Mbits/sec  0.043 ms  2416/65488 (3.7%)

[  5]   9.00-10.00  sec  94.6 MBytes   794 Mbits/sec  0.011 ms  3219/71756 (4.5%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec   935 MBytes   784 Mbits/sec  0.000 ms  0/677200 (0%)  sender

[  5]   0.00-10.00  sec   915 MBytes   767 Mbits/sec  0.011 ms  14572/677101 (2.2%)  receiver



iperf Done.

joe@ubuntuserver:~$





My network interfaces

Code: 
root@pve:/etc/network# cat interfaces



auto lo

iface lo inet loopback



auto eno1

iface eno1 inet manual



auto eno2

iface eno2 inet manual



auto ens1f0

iface ens1f0 inet manual



auto ens1f1

iface ens1f1 inet manual



auto bond0

iface bond0 inet manual

        bond-slaves ens1f0 ens1f1

        bond-miimon 100

        bond-mode 802.3ad

        bond-xmit-hash-policy layer2+3



auto vmbr0

iface vmbr0 inet manual

        bridge-ports bond0

        bridge-stp off

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 44 35 192 172 999 888



auto vmbr0.35

        iface vmbr0.35 inet static

        address 10.44.35.4/24

        gateway 10.44.35.1



auto vmbr1

iface vmbr1 inet manual

        bridge-ports eno2

        bridge-stp off

        bridge-fd 0

        bridge-vlan-aware yes

        bridge-vids 2-4094





I then tested using the ubuntu desktop instance as the iperf3 server, with the Proxmox host connecting as client. Results



Code: 
root@pve:/etc/network# iperf3 -b 0 -u -Z -c 10.44.35.128

Connecting to host 10.44.35.128, port 5201

[  5] local 10.44.35.4 port 34457 connected to 10.44.35.128 port 5201

[ ID] Interval           Transfer     Bitrate         Total Datagrams

[  5]   0.00-1.00   sec   211 MBytes  1.77 Gbits/sec  152820

[  5]   1.00-2.00   sec   245 MBytes  2.05 Gbits/sec  177340

[  5]   2.00-3.00   sec   247 MBytes  2.07 Gbits/sec  179060

[  5]   3.00-4.00   sec   253 MBytes  2.13 Gbits/sec  183530

[  5]   4.00-5.00   sec   251 MBytes  2.10 Gbits/sec  181490

[  5]   5.00-6.00   sec   237 MBytes  1.98 Gbits/sec  171330

[  5]   6.00-7.00   sec   236 MBytes  1.98 Gbits/sec  170610

[  5]   7.00-8.00   sec   239 MBytes  2.00 Gbits/sec  172750

[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec  172670

[  5]   9.00-10.00  sec   242 MBytes  2.03 Gbits/sec  174980

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datag                                     rams

[  5]   0.00-10.00  sec  2.34 GBytes  2.01 Gbits/sec  0.000 ms  0/1736580 (0%)                                       sender

[  5]   0.00-10.00  sec   790 MBytes   663 Mbits/sec  0.009 ms  1164324/1736480                                      (67%)  receiver



iperf Done.

root@pve:/etc/network# iperf3 -b 0 -Z -c 10.44.35.128

Connecting to host 10.44.35.128, port 5201

[  5] local 10.44.35.4 port 60050 connected to 10.44.35.128 port 5201

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

[  5]   0.00-1.00   sec  1.87 GBytes  16.0 Gbits/sec    0   1.31 MBytes

[  5]   1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec    0   1.75 MBytes

[  5]   2.00-3.00   sec  1.82 GBytes  15.7 Gbits/sec    0   2.28 MBytes

[  5]   3.00-4.00   sec  1.82 GBytes  15.6 Gbits/sec    0   2.91 MBytes

[  5]   4.00-5.00   sec  1.87 GBytes  16.1 Gbits/sec    0   2.91 MBytes

[  5]   5.00-6.00   sec  1.92 GBytes  16.5 Gbits/sec    0   3.05 MBytes

[  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec    0   3.05 MBytes

[  5]   7.00-8.00   sec  1.68 GBytes  14.4 Gbits/sec    0   3.05 MBytes

[  5]   8.00-9.00   sec  1.94 GBytes  16.7 Gbits/sec    0   3.05 MBytes

[  5]   9.00-10.00  sec  1.86 GBytes  16.0 Gbits/sec    0   3.05 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Retr

[  5]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec    0             sender

[  5]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec                  receiver



iperf Done.





Any ideas?
 
Last edited:
Another test of UDP from Proxmox Host to VM

When VM is sending to Proxmox host: 900Mbps, 0.12% packet loss
When Proxmox is sending to VM: 2.09 Gbps, 79% packet loss


Code: 
root@pve:/etc/network# iperf3 -b 0 -u -R -c 10.44.35.102
Connecting to host 10.44.35.102, port 5201
Reverse mode, remote host 10.44.35.102 is sending
[  5] local 10.44.35.4 port 46246 connected to 10.44.35.102 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   105 MBytes   882 Mbits/sec  0.006 ms  0/76151 (0%)
[  5]   1.00-2.00   sec   109 MBytes   913 Mbits/sec  0.006 ms  79/78854 (0.1%)
[  5]   2.00-3.00   sec   107 MBytes   901 Mbits/sec  0.004 ms  0/77744 (0%)
[  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec  0.002 ms  627/74273 (0.84%)
[  5]   4.00-5.00   sec   109 MBytes   918 Mbits/sec  0.007 ms  0/79252 (0%)
[  5]   5.00-6.00   sec   102 MBytes   854 Mbits/sec  0.005 ms  0/73728 (0%)
[  5]   6.00-7.00   sec   108 MBytes   909 Mbits/sec  0.004 ms  0/78499 (0%)
[  5]   7.00-8.00   sec   109 MBytes   915 Mbits/sec  0.003 ms  213/79235 (0.27%)
[  5]   8.00-9.00   sec   110 MBytes   919 Mbits/sec  0.005 ms  0/79338 (0%)
[  5]   9.00-10.00  sec   114 MBytes   959 Mbits/sec  0.003 ms  0/82787 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.05 GBytes   903 Mbits/sec  0.000 ms  0/779870 (0%)  sender
[  5]   0.00-10.00  sec  1.05 GBytes   902 Mbits/sec  0.003 ms  919/779861 (0.12%)  receiver

iperf Done.
root@pve:/etc/network# iperf3 -b 0 -u -c 10.44.35.102
Connecting to host 10.44.35.102, port 5201
[  5] local 10.44.35.4 port 32846 connected to 10.44.35.102 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   250 MBytes  2.09 Gbits/sec  180840
[  5]   1.00-2.00   sec   247 MBytes  2.07 Gbits/sec  178910
[  5]   2.00-3.00   sec   245 MBytes  2.05 Gbits/sec  177110
[  5]   3.00-4.00   sec   251 MBytes  2.10 Gbits/sec  181700
[  5]   4.00-5.00   sec   257 MBytes  2.16 Gbits/sec  186460
[  5]   5.00-6.00   sec   246 MBytes  2.06 Gbits/sec  178150
[  5]   6.00-7.00   sec   243 MBytes  2.04 Gbits/sec  176190
[  5]   7.00-8.00   sec   251 MBytes  2.11 Gbits/sec  182120
[  5]   8.00-9.00   sec   229 MBytes  1.92 Gbits/sec  165570
[  5]   9.00-10.00  sec   241 MBytes  2.02 Gbits/sec  174730
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.40 GBytes  2.06 Gbits/sec  0.000 ms  0/1781780 (0%)  sender
[  5]   0.00-10.00  sec   519 MBytes   435 Mbits/sec  0.023 ms  1405660/1781614 (79%)  receiver

iperf Done.
root@pve:/etc/network#
 
Can anyone offer advice? What other info do you need see to help me understand what is going on ?
The hardware is, for host, 1x E5-2630v3, 48GB memory, but I am not complaining about TCP speeds.
 
You must log in or register to reply here.
Top Bottom