[SOLVED] VirtIO Windows Network Very Slow

D

Darren

New Member
Jul 28, 2016
21
1
1
47
I have a bunch of Windows 2008R2 and 2012R2 servers on Proxmox using ZFS and VirtIO. Using CrystalDiskMark with the 8Gbyte test file setting, I can get over 2 Gbytes/sec for sequential read and write and over 100,000 QD32 IOPS. The network is bridged (EDIT: I had nat'ed here but it is obviously not) through a single Intel T540 10Gb ethernet connection. If I try to transfer through the network to the Windows VM guests, sometimes I can get 1.2 Gbytes/sec for a short period of time and then it bounces around between a few megabytes/sec to some tens of megabytes/sec. Usually, I don't see more than 30MB/s.

Here's a default iperf:

Code: 
C:\>iperf3 -p 5201 -c 10.10.20.24
Connecting to host 10.10.20.24, port 5201
[  4] local 10.10.20.22 port 50579 connected to 10.10.20.24 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   249 MBytes  2.09 Gbits/sec
[  4]   1.00-2.00   sec   264 MBytes  2.21 Gbits/sec
[  4]   2.00-3.00   sec   293 MBytes  2.45 Gbits/sec
[  4]   3.00-4.00   sec   286 MBytes  2.40 Gbits/sec
[  4]   4.00-4.36   sec  94.5 MBytes  2.23 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-4.36   sec  1.16 GBytes  2.28 Gbits/sec                  sender
[  4]   0.00-4.36   sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

However, when I throw 20 sessions in parallel, this is what happens:

Code: 
C:\>iperf3 -P 20 -p 5201 -c 10.10.20.24
Connecting to host 10.10.20.24, port 5201
[  4] local 10.10.20.22 port 50587 connected to 10.10.20.24 port 5201
[  6] local 10.10.20.22 port 50588 connected to 10.10.20.24 port 5201
[  8] local 10.10.20.22 port 50589 connected to 10.10.20.24 port 5201
[ 10] local 10.10.20.22 port 50590 connected to 10.10.20.24 port 5201
[ 12] local 10.10.20.22 port 50591 connected to 10.10.20.24 port 5201
[ 14] local 10.10.20.22 port 50592 connected to 10.10.20.24 port 5201
[ 16] local 10.10.20.22 port 50593 connected to 10.10.20.24 port 5201
[ 18] local 10.10.20.22 port 50594 connected to 10.10.20.24 port 5201
[ 20] local 10.10.20.22 port 50595 connected to 10.10.20.24 port 5201
[ 22] local 10.10.20.22 port 50596 connected to 10.10.20.24 port 5201
[ 24] local 10.10.20.22 port 50597 connected to 10.10.20.24 port 5201
[ 26] local 10.10.20.22 port 50598 connected to 10.10.20.24 port 5201
[ 28] local 10.10.20.22 port 50599 connected to 10.10.20.24 port 5201
[ 30] local 10.10.20.22 port 50600 connected to 10.10.20.24 port 5201
[ 32] local 10.10.20.22 port 50601 connected to 10.10.20.24 port 5201
[ 34] local 10.10.20.22 port 50602 connected to 10.10.20.24 port 5201
[ 36] local 10.10.20.22 port 50603 connected to 10.10.20.24 port 5201
[ 38] local 10.10.20.22 port 50604 connected to 10.10.20.24 port 5201
[ 40] local 10.10.20.22 port 50605 connected to 10.10.20.24 port 5201
[ 42] local 10.10.20.22 port 50606 connected to 10.10.20.24 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[  6]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[  8]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[ 10]   0.00-1.00   sec   640 KBytes  5.24 Mbits/sec
[ 12]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 14]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 16]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 18]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec
[ 20]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 22]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 24]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 26]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 28]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 30]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 32]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 34]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 36]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 38]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 40]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[ 42]   0.00-1.00   sec   384 KBytes  3.14 Mbits/sec
[SUM]   0.00-1.00   sec  8.88 MBytes  74.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  6]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  8]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 10]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 12]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 14]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 16]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 18]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 20]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 22]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 24]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 26]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 28]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 30]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 32]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 34]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 36]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 38]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 40]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[ 42]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[SUM]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
iperf3: error - unable to receive control message: Connection reset by peer

This happens during the course of regular use at times - disconnecting and then reconnecting the virtual NIC repairs the server side. I've spent over a year fiddling with this to no avail.

Please help.
 
Last edited:
Here's ifconfig output. Notice the dropped packets on eth0 (10Gb t540 NIC) and the vmbr0:

Code: 
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 94619700  bytes 133137407682 (123.9 GiB)
        RX errors 0  dropped 1780  overruns 0  frame 0
        TX packets 103573528  bytes 132267899900 (123.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0.20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 93910155  bytes 131519293025 (122.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 103312689  bytes 132195192641 (123.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 123751  bytes 36660081 (34.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 123751  bytes 36660081 (34.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap101i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 12:51:4b:7d:dc:fd  txqueuelen 1000  (Ethernet)
        RX packets 2388495  bytes 1234280187 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4750807  bytes 4856053215 (4.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap102i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 72:86:9a:fe:f0:d3  txqueuelen 1000  (Ethernet)
        RX packets 831186  bytes 274921869 (262.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1581366  bytes 1703700216 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap103i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 66:8f:bc:0c:62:41  txqueuelen 1000  (Ethernet)
        RX packets 7895345  bytes 3742289494 (3.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10330730  bytes 7021440894 (6.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap104i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 82:a1:c6:d8:95:a0  txqueuelen 1000  (Ethernet)
        RX packets 972742  bytes 456969790 (435.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1522611  bytes 1389811929 (1.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap105i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether be:2a:d1:b7:e3:6b  txqueuelen 1000  (Ethernet)
        RX packets 241687  bytes 73185976 (69.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 354628  bytes 203018619 (193.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap106i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 6e:52:13:8b:c2:ac  txqueuelen 1000  (Ethernet)
        RX packets 23578608  bytes 4759140171 (4.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 129251372  bytes 184767893603 (172.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap107i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether ca:d8:ee:7c:c1:8b  txqueuelen 1000  (Ethernet)
        RX packets 128978582  bytes 182977092995 (170.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 103823428  bytes 142121209913 (132.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap108i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 1a:74:bd:6a:08:a0  txqueuelen 1000  (Ethernet)
        RX packets 7827613  bytes 1302610226 (1.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7938651  bytes 2631302386 (2.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap109i0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether fa:7d:59:b0:30:0b  txqueuelen 1000  (Ethernet)
        RX packets 123898291  bytes 189449928317 (176.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27938113  bytes 41190923759 (38.3 GiB)
        TX errors 0  dropped 781 overruns 0  carrier 0  collisions 0

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.0.41  netmask 255.255.254.0  broadcast 10.10.1.255
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 644275  bytes 290448809 (276.9 MiB)
        RX errors 0  dropped 10675  overruns 0  frame 0
        TX packets 76850  bytes 62530653 (59.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vmbr0v20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:6f:21:10  txqueuelen 1000  (Ethernet)
        RX packets 31777  bytes 3258930 (3.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
this sounds like a low-level network problem; I recommend upgrading your NIC firmware and upgrading your NIC kernel module (Intel latest tends to be much newer than what is in the kernel by default) and then re-checking all your iperf3 performance. If you can't get reliable / reproducible iperf3 results, you're not going to be able to rely on anything in higher layers.
 
This was solved by hanging the Proxmox VMs off of Intel SR-IOV Virtual Function adapters instead of a direct bridge at the Ethernet level.

https://www.intel.com/content/dam/w...s/technology-briefs/sr-iov-nfv-tech-brief.pdf

It would be nice if there were a proper GUI config for all of this fiasco (ixgbe.conf; max_vfs=16,16; modprobe vfio; update-grub; etc). In the many hours that I spent working through this, I've stumbled across content which reveals that there are at least a few others who have experienced the same issues.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back