[SOLVED] VirtIO Windows Network Very Slow

D

Darren

New Member
Jul 28, 2016
21
1
1
47
I have a bunch of Windows 2008R2 and 2012R2 servers on Proxmox using ZFS and VirtIO. Using CrystalDiskMark with the 8Gbyte test file setting, I can get over 2 Gbytes/sec for sequential read and write and over 100,000 QD32 IOPS. The network is bridged (EDIT: I had nat'ed here but it is obviously not) through a single Intel T540 10Gb ethernet connection. If I try to transfer through the network to the Windows VM guests, sometimes I can get 1.2 Gbytes/sec for a short period of time and then it bounces around between a few megabytes/sec to some tens of megabytes/sec. Usually, I don't see more than 30MB/s.

Here's a default iperf:

Code: 
C:\>iperf3 -p 5201 -c 10.10.20.24
Connecting to host 10.10.20.24, port 5201
[  4] local 10.10.20.22 port 50579 connected to 10.10.20.24 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   249 MBytes  2.09 Gbits/sec
[  4]   1.00-2.00   sec   264 MBytes  2.21 Gbits/sec
[  4]   2.00-3.00   sec   293 MBytes  2.45 Gbits/sec
[  4]   3.00-4.00   sec   286 MBytes  2.40 Gbits/sec
[  4]   4.00-4.36   sec  94.5 MBytes  2.23 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-4.36   sec  1.16 GBytes  2.28 Gbits/sec                  sender
[  4]   0.00-4.36   sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

However, when I throw 20 sessions in parallel, this is what happens:

Code: 
C:\>iperf3 -P 20 -p 5201 -c 10.10.20.24
Connecting to host 10.10.20.24, port 5201
[  4] local 10.10.20.22 port 50587 connected to 10.10.20.24 port 5201
[  6] local 10.10.20.22 port 50588 connected to 10.10.20.24 port 5201
[  8] local 10.10.20.22 port 50589 connected to 10.10.20.24 port 5201
[ 10] local 10.10.20.22 port 50590 connected to 10.10.20.24 port 5201
[ 12] local 10.10.20.22 port 50591 connected to 10.10.20.24 port 5201
[ 14] local 10.10.20.22 port 50592 connected to 10.10.20.24 port 5201
[ 16] local 10.10.20.22 port 50593 connected to 10.10.20.24 port 5201
[ 18] local 10.10.20.22 port 50594 connected to 10.10.20.24 port 5201
[ 20] local 10.10.20.22 port 50595 connected to 10.10.20.24 port 5201
[ 22] local 10.10.20.22 port 50596 connected to 10.10.20.24 port 5201
[ 24] local 10.10.20.22 port 50597 connected to 10.10.20.24 port 5201
[ 26] local 10.10.20.22 port 50598 connected to 10.10.20.24 port 5201
[ 28] local 10.10.20.22 port 50599 connected to 10.10.20.24 port 5201
[ 30] local 10.10.20.22 port 50600 connected to 10.10.20.24 port 5201
[ 32] local 10.10.20.22 port 50601 connected to 10.10.20.24 port 5201
[ 34] local 10.10.20.22 port 50602 connected to 10.10.20.24 port 5201
[ 36] local 10.10.20.22 port 50603 connected to 10.10.20.24 port 5201
[ 38] local 10.10.20.22 port 50604 connected to 10.10.20.24 port 5201
[ 40] local 10.10.20.22 port 50605 connected to 10.10.20.24 port 5201
[ 42] local 10.10.20.22 port 50606 connected to 10.10.20.24 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[  6]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[  8]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[ 10]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[ 12]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 14]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 16]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 18]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 20]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 22]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 24]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 26]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 28]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 30]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 32]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 34]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 36]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 38]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 40]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 42]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[SUM]   0.00-1.00   sec  8.88 MBytes  74.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
iperf3: error - unable to receive control message: Connection reset by peer

This happens during the course of regular use at times - disconnecting and then reconnecting the virtual NIC repairs the server side. I've spent over a year fiddling with this to no avail.

Please help.
 
Last edited:
Here's ifconfig output. Notice the dropped packets on eth0 (10Gb t540 NIC) and the vmbr0:

Code: 
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 94619700  bytes 133137407682 (123.9 GiB)
        RX errors 0  dropped 1780  overruns 0  frame 0
        TX packets 103573528  bytes 132267899900 (123.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0.20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 93910155  bytes 131519293025 (122.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 103312689  bytes 132195192641 (123.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 123751  bytes 36660081 (34.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 123751  bytes 36660081 (34.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap101i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 12:51:4b:7d:dc:fd  txqueuelen 1000  (Ethernet)
        RX packets 2388495  bytes 1234280187 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4750807  bytes 4856053215 (4.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap102i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 72:86:9a:fe:f0:d3  txqueuelen 1000  (Ethernet)
        RX packets 831186  bytes 274921869 (262.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1581366  bytes 1703700216 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap103i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 66:8f:bc:0c:62:41  txqueuelen 1000  (Ethernet)
        RX packets 7895345  bytes 3742289494 (3.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10330730  bytes 7021440894 (6.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap104i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 82:a1:c6:d8:95:a0  txqueuelen 1000  (Ethernet)
        RX packets 972742  bytes 456969790 (435.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1522611  bytes 1389811929 (1.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap105i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether be:2a:d1:b7:e3:6b  txqueuelen 1000  (Ethernet)
        RX packets 241687  bytes 73185976 (69.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 354628  bytes 203018619 (193.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap106i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 6e:52:13:8b:c2:ac  txqueuelen 1000  (Ethernet)
        RX packets 23578608  bytes 4759140171 (4.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 129251372  bytes 184767893603 (172.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap107i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether ca:d8:ee:7c:c1:8b  txqueuelen 1000  (Ethernet)
        RX packets 128978582  bytes 182977092995 (170.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 103823428  bytes 142121209913 (132.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap108i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 1a:74:bd:6a:08:a0  txqueuelen 1000  (Ethernet)
        RX packets 7827613  bytes 1302610226 (1.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7938651  bytes 2631302386 (2.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap109i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether fa:7d:59:b0:30:0b  txqueuelen 1000  (Ethernet)
        RX packets 123898291  bytes 189449928317 (176.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27938113  bytes 41190923759 (38.3 GiB)
        TX errors 0  dropped 781 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.0.41  netmask 255.255.254.0  broadcast 10.10.1.255
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 644275  bytes 290448809 (276.9 MiB)
        RX errors 0  dropped 10675  overruns 0  frame 0
        TX packets 76850  bytes 62530653 (59.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0v20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 31777  bytes 3258930 (3.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
this sounds like a low-level network problem; I recommend upgrading your NIC firmware and upgrading your NIC kernel module (Intel latest tends to be much newer than what is in the kernel by default) and then re-checking all your iperf3 performance. If you can't get reliable / reproducible iperf3 results, you're not going to be able to rely on anything in higher layers.
 
This was solved by hanging the Proxmox VMs off of Intel SR-IOV Virtual Function adapters instead of a direct bridge at the Ethernet level.

https://www.intel.com/content/dam/w...s/technology-briefs/sr-iov-nfv-tech-brief.pdf

It would be nice if there were a proper GUI config for all of this fiasco (ixgbe.conf; max_vfs=16,16; modprobe vfio; update-grub; etc). In the many hours that I spent working through this, I've stumbled across content which reveals that there are at least a few others who have experienced the same issues.
 
Last edited:
You must log in or register to reply here.
Top Bottom