Hi Guys,
I am looking into some network issues i am having. On the daily use, no issues are noticable but when performing speedtests, I can clearly see speeds are dropping down to 300mbit/sec on gbit lines.
Windows VM(virtio 10G NIC) to Windows VM(virtio 10G NIC) routed via PFsense VM (10G) on the same PVE host. Same results both ways.;
Connecting to host 192.168.30.50, port 5201
[ 4] local 192.168.20.50 port 55810 connected to 192.168.30.50 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 157 MBytes 1.32 Gbits/sec
[ 4] 1.00-2.00 sec 177 MBytes 1.49 Gbits/sec
[ 4] 2.00-3.00 sec 197 MBytes 1.65 Gbits/sec
[ 4] 3.00-4.00 sec 191 MBytes 1.60 Gbits/sec
[ 4] 4.00-5.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 5.00-6.00 sec 204 MBytes 1.71 Gbits/sec
[ 4] 6.00-7.00 sec 200 MBytes 1.68 Gbits/sec
[ 4] 7.00-8.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 8.00-9.00 sec 186 MBytes 1.56 Gbits/sec
[ 4] 9.00-10.00 sec 195 MBytes 1.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec receiver
Windows VM (virtuio 10G NIC) to Linux LXC (uses 2x1G LACP of host i guess). On different PVE hosts (both hosts LACP connected to switch). Performance OK both ways.
Accepted connection from 192.168.20.30, port 44724
[ 5] local 192.168.20.50 port 5201 connected to 192.168.20.30 port 44726
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 94.8 MBytes 795 Mbits/sec
[ 5] 1.00-2.00 sec 96.3 MBytes 808 Mbits/sec
[ 5] 2.00-3.00 sec 95.8 MBytes 804 Mbits/sec
[ 5] 3.00-4.00 sec 99.0 MBytes 831 Mbits/sec
[ 5] 4.00-5.00 sec 93.9 MBytes 787 Mbits/sec
[ 5] 5.00-6.00 sec 97.5 MBytes 818 Mbits/sec
[ 5] 6.00-7.00 sec 96.2 MBytes 807 Mbits/sec
[ 5] 7.00-8.00 sec 97.3 MBytes 817 Mbits/sec
[ 5] 8.00-9.00 sec 98.6 MBytes 827 Mbits/sec
[ 5] 9.00-10.00 sec 99.0 MBytes 831 Mbits/sec
[ 5] 10.00-10.02 sec 2.28 MBytes 861 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.02 sec 971 MBytes 812 Mbits/sec receiver
So; inside the VM's and from PVE host to PVE host, it seems to perform fine.
Now i'm switching to testing on the PVE hosts themselves;
PVE host 1 to PVE host 2 (both 2x 1G LACP connected to switch)
Connecting to host 10.10.10.202, port 5201
[ 4] local 10.10.10.201 port 42428 connected to 10.10.10.202 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 105 MBytes 884 Mbits/sec 121 298 KBytes
[ 4] 1.00-2.00 sec 107 MBytes 896 Mbits/sec 107 67.9 KBytes
[ 4] 2.00-3.00 sec 105 MBytes 883 Mbits/sec 151 67.9 KBytes
[ 4] 3.00-4.00 sec 103 MBytes 865 Mbits/sec 219 52.3 KBytes
[ 4] 4.00-5.00 sec 107 MBytes 894 Mbits/sec 161 72.1 KBytes
[ 4] 5.00-6.00 sec 103 MBytes 861 Mbits/sec 151 199 KBytes
[ 4] 6.00-7.00 sec 107 MBytes 897 Mbits/sec 137 286 KBytes
[ 4] 7.00-8.00 sec 106 MBytes 888 Mbits/sec 154 43.8 KBytes
[ 4] 8.00-9.00 sec 104 MBytes 876 Mbits/sec 174 185 KBytes
[ 4] 9.00-10.00 sec 106 MBytes 888 Mbits/sec 183 73.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.03 GBytes 883 Mbits/sec 1558 sender
[ 4] 0.00-10.00 sec 1.03 GBytes 882 Mbits/sec receiver
Here we see issues are starting to creep in. While speeds are ok, iperf3 reports retransmissions when testing. Both ways.
Next on to testing to my new PFsense firewall which is connected with 2x 10G LACP to the switch;
Firewall as server, PVE host as client;
Connecting to host 10.10.10.254, port 5201
[ 4] local 10.10.10.201 port 57174 connected to 10.10.10.254 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 957 Mbits/sec 0 387 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 0 387 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 404 KBytes
[ 4] 3.00-4.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 426 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec receiver
PVE host as server, Firewall as clients;
Connecting to host 10.10.10.201, port 5201
[ 5] local 10.10.10.254 port 52831 connected to 10.10.10.201 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 51.0 MBytes 428 Mbits/sec 258 12.8 KBytes
[ 5] 1.00-2.00 sec 50.9 MBytes 427 Mbits/sec 260 14.2 KBytes
[ 5] 2.00-3.00 sec 52.2 MBytes 438 Mbits/sec 335 9.92 KBytes
[ 5] 3.00-4.00 sec 51.8 MBytes 434 Mbits/sec 329 9.92 KBytes
[ 5] 4.00-5.00 sec 52.2 MBytes 438 Mbits/sec 342 8.51 KBytes
[ 5] 5.00-6.00 sec 51.2 MBytes 429 Mbits/sec 297 11.3 KBytes
[ 5] 6.00-7.00 sec 51.2 MBytes 430 Mbits/sec 310 9.93 KBytes
[ 5] 7.00-8.00 sec 51.3 MBytes 430 Mbits/sec 308 8.50 KBytes
[ 5] 8.00-9.00 sec 51.5 MBytes 432 Mbits/sec 326 14.2 KBytes
[ 5] 9.00-10.00 sec 50.2 MBytes 421 Mbits/sec 299 15.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 514 MBytes 431 Mbits/sec 3064 sender
[ 5] 0.00-10.00 sec 514 MBytes 431 Mbits/sec receiver
Even more retries and speeds are suffering.
All PVE nodes are also connected via 10G network for Ceph only via the same switch.
Connecting to host 192.168.1.2, port 5201
[ 4] local 192.168.1.1 port 43338 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.04 GBytes 8.97 Gbits/sec 106 549 KBytes
[ 4] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 5 646 KBytes
[ 4] 2.00-3.00 sec 1.08 GBytes 9.31 Gbits/sec 25 542 KBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.34 Gbits/sec 45 570 KBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec 4 214 KBytes
[ 4] 5.00-6.00 sec 1.02 GBytes 8.77 Gbits/sec 122 694 KBytes
[ 4] 6.00-7.00 sec 1.08 GBytes 9.32 Gbits/sec 10 577 KBytes
[ 4] 7.00-8.00 sec 1.07 GBytes 9.21 Gbits/sec 47 543 KBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.38 Gbits/sec 0 601 KBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 6 578 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.8 GBytes 9.25 Gbits/sec 370 sender
[ 4] 0.00-10.00 sec 10.8 GBytes 9.24 Gbits/sec receiver
I'm using Proxmox 5.3.5, Supermicro hardware with Intel 82576 NIC's (driver version 5.3.5.18)
The PFsense box has Chelsio 10G NIC's.
The switch is a Mikrotik CRS317-1G-16S+. No errors on ports.
I guess some retransmissions is ok as long as link is saturated at that point.
Is this a typical behaviour of problems with 10G->1G or 1G->10G flows?
Is the 10G node requesting so much data that the 1G side needs to drop packets and kill performance?
The real reason behind the troubleshooting is the internet performance in a VM.
Speedtest on the firewall itsself is reaching over 5Gbit/sec from internet speedtest.
Speedtest that same file in a VM and speeds drop to the ~300mbit down as seen above. Upload speed however is just fine at 900mbit.
I am looking into some network issues i am having. On the daily use, no issues are noticable but when performing speedtests, I can clearly see speeds are dropping down to 300mbit/sec on gbit lines.
Windows VM(virtio 10G NIC) to Windows VM(virtio 10G NIC) routed via PFsense VM (10G) on the same PVE host. Same results both ways.;
Connecting to host 192.168.30.50, port 5201
[ 4] local 192.168.20.50 port 55810 connected to 192.168.30.50 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 157 MBytes 1.32 Gbits/sec
[ 4] 1.00-2.00 sec 177 MBytes 1.49 Gbits/sec
[ 4] 2.00-3.00 sec 197 MBytes 1.65 Gbits/sec
[ 4] 3.00-4.00 sec 191 MBytes 1.60 Gbits/sec
[ 4] 4.00-5.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 5.00-6.00 sec 204 MBytes 1.71 Gbits/sec
[ 4] 6.00-7.00 sec 200 MBytes 1.68 Gbits/sec
[ 4] 7.00-8.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 8.00-9.00 sec 186 MBytes 1.56 Gbits/sec
[ 4] 9.00-10.00 sec 195 MBytes 1.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.84 GBytes 1.58 Gbits/sec receiver
Windows VM (virtuio 10G NIC) to Linux LXC (uses 2x1G LACP of host i guess). On different PVE hosts (both hosts LACP connected to switch). Performance OK both ways.
Accepted connection from 192.168.20.30, port 44724
[ 5] local 192.168.20.50 port 5201 connected to 192.168.20.30 port 44726
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 94.8 MBytes 795 Mbits/sec
[ 5] 1.00-2.00 sec 96.3 MBytes 808 Mbits/sec
[ 5] 2.00-3.00 sec 95.8 MBytes 804 Mbits/sec
[ 5] 3.00-4.00 sec 99.0 MBytes 831 Mbits/sec
[ 5] 4.00-5.00 sec 93.9 MBytes 787 Mbits/sec
[ 5] 5.00-6.00 sec 97.5 MBytes 818 Mbits/sec
[ 5] 6.00-7.00 sec 96.2 MBytes 807 Mbits/sec
[ 5] 7.00-8.00 sec 97.3 MBytes 817 Mbits/sec
[ 5] 8.00-9.00 sec 98.6 MBytes 827 Mbits/sec
[ 5] 9.00-10.00 sec 99.0 MBytes 831 Mbits/sec
[ 5] 10.00-10.02 sec 2.28 MBytes 861 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.02 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.02 sec 971 MBytes 812 Mbits/sec receiver
So; inside the VM's and from PVE host to PVE host, it seems to perform fine.
Now i'm switching to testing on the PVE hosts themselves;
PVE host 1 to PVE host 2 (both 2x 1G LACP connected to switch)
Connecting to host 10.10.10.202, port 5201
[ 4] local 10.10.10.201 port 42428 connected to 10.10.10.202 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 105 MBytes 884 Mbits/sec 121 298 KBytes
[ 4] 1.00-2.00 sec 107 MBytes 896 Mbits/sec 107 67.9 KBytes
[ 4] 2.00-3.00 sec 105 MBytes 883 Mbits/sec 151 67.9 KBytes
[ 4] 3.00-4.00 sec 103 MBytes 865 Mbits/sec 219 52.3 KBytes
[ 4] 4.00-5.00 sec 107 MBytes 894 Mbits/sec 161 72.1 KBytes
[ 4] 5.00-6.00 sec 103 MBytes 861 Mbits/sec 151 199 KBytes
[ 4] 6.00-7.00 sec 107 MBytes 897 Mbits/sec 137 286 KBytes
[ 4] 7.00-8.00 sec 106 MBytes 888 Mbits/sec 154 43.8 KBytes
[ 4] 8.00-9.00 sec 104 MBytes 876 Mbits/sec 174 185 KBytes
[ 4] 9.00-10.00 sec 106 MBytes 888 Mbits/sec 183 73.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.03 GBytes 883 Mbits/sec 1558 sender
[ 4] 0.00-10.00 sec 1.03 GBytes 882 Mbits/sec receiver
Here we see issues are starting to creep in. While speeds are ok, iperf3 reports retransmissions when testing. Both ways.
Next on to testing to my new PFsense firewall which is connected with 2x 10G LACP to the switch;
Firewall as server, PVE host as client;
Connecting to host 10.10.10.254, port 5201
[ 4] local 10.10.10.201 port 57174 connected to 10.10.10.254 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 957 Mbits/sec 0 387 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 0 387 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 404 KBytes
[ 4] 3.00-4.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 404 KBytes
[ 4] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 426 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 447 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec receiver
PVE host as server, Firewall as clients;
Connecting to host 10.10.10.201, port 5201
[ 5] local 10.10.10.254 port 52831 connected to 10.10.10.201 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 51.0 MBytes 428 Mbits/sec 258 12.8 KBytes
[ 5] 1.00-2.00 sec 50.9 MBytes 427 Mbits/sec 260 14.2 KBytes
[ 5] 2.00-3.00 sec 52.2 MBytes 438 Mbits/sec 335 9.92 KBytes
[ 5] 3.00-4.00 sec 51.8 MBytes 434 Mbits/sec 329 9.92 KBytes
[ 5] 4.00-5.00 sec 52.2 MBytes 438 Mbits/sec 342 8.51 KBytes
[ 5] 5.00-6.00 sec 51.2 MBytes 429 Mbits/sec 297 11.3 KBytes
[ 5] 6.00-7.00 sec 51.2 MBytes 430 Mbits/sec 310 9.93 KBytes
[ 5] 7.00-8.00 sec 51.3 MBytes 430 Mbits/sec 308 8.50 KBytes
[ 5] 8.00-9.00 sec 51.5 MBytes 432 Mbits/sec 326 14.2 KBytes
[ 5] 9.00-10.00 sec 50.2 MBytes 421 Mbits/sec 299 15.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 514 MBytes 431 Mbits/sec 3064 sender
[ 5] 0.00-10.00 sec 514 MBytes 431 Mbits/sec receiver
Even more retries and speeds are suffering.
All PVE nodes are also connected via 10G network for Ceph only via the same switch.
Connecting to host 192.168.1.2, port 5201
[ 4] local 192.168.1.1 port 43338 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.04 GBytes 8.97 Gbits/sec 106 549 KBytes
[ 4] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 5 646 KBytes
[ 4] 2.00-3.00 sec 1.08 GBytes 9.31 Gbits/sec 25 542 KBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.34 Gbits/sec 45 570 KBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec 4 214 KBytes
[ 4] 5.00-6.00 sec 1.02 GBytes 8.77 Gbits/sec 122 694 KBytes
[ 4] 6.00-7.00 sec 1.08 GBytes 9.32 Gbits/sec 10 577 KBytes
[ 4] 7.00-8.00 sec 1.07 GBytes 9.21 Gbits/sec 47 543 KBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.38 Gbits/sec 0 601 KBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 6 578 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.8 GBytes 9.25 Gbits/sec 370 sender
[ 4] 0.00-10.00 sec 10.8 GBytes 9.24 Gbits/sec receiver
I'm using Proxmox 5.3.5, Supermicro hardware with Intel 82576 NIC's (driver version 5.3.5.18)
The PFsense box has Chelsio 10G NIC's.
The switch is a Mikrotik CRS317-1G-16S+. No errors on ports.
I guess some retransmissions is ok as long as link is saturated at that point.
Is this a typical behaviour of problems with 10G->1G or 1G->10G flows?
Is the 10G node requesting so much data that the 1G side needs to drop packets and kill performance?
The real reason behind the troubleshooting is the internet performance in a VM.
Speedtest on the firewall itsself is reaching over 5Gbit/sec from internet speedtest.
Speedtest that same file in a VM and speeds drop to the ~300mbit down as seen above. Upload speed however is just fine at 900mbit.
Last edited: