Slow LAN speeds on gigabit NICs

[manf0001]

Hello, I'm running Proxmox VE 5.0-23/af4267bf the host hardware specs are:

-HP Proliant D380 G6, with 2 Intel Xeon X5570 (2 Sockets 8 core each)
-Four gigabit NICs.
-16 GB of RAM
- Five 4TB HD's setup in a RAID 5 to get 16TB of space.

I have Vmbr0 which is the main LAN connection going into my switch and VMbr02 which has one NIC that I'm using as a point to point transfer with a NAS.

I've setup a VM with CentOS 7 with Kernel 3.10.0-514.26.2.el7.x86_64, with the two NICs as stated above.
I have a 100GB hard disk for my main install, and a 10TB disk for my data (lots and lots of media)

The main NIC is working fine for accessing the network and internet, I've configured NIC2 with a static IP address different then my main network, with a MTU of 9000.

The NAS whose Data I"m trying to copy over has an IP address of the same subnet of NIC2 on my cent server also with an MTU of 9000. The NAS also has a SAMBA share.

From my Cent Server I can connect to the share but when I try to copy large amounts of data my speeds are very low it starts at about 25MB/s but drops to about 4-5MB/s.

Am I missing something here? because when I attach my laptop running Fedora, and do the same thing with the NAS, I can transfer files at over 100MB/s speeds.

As a test I installed Fedora as a VM but got the same speeds as my cent.

- I'm running Virtio network devices, have tried switching between the others.
-I've disabled the firewall and SELinux on the VM,
-Even turned off the firewall on the Proxmox host.
I'm not sure if the HD cache has any bearing but it's set as the Default (No Cache)

TIA

 

[Richard]

From my Cent Server I can connect to the share but when I try to copy large amounts of data my speeds are very low it starts at about 25MB/s but drops to about 4-5MB/s.

As a first step it's recommended to make some tests in order to figure out where the bottleneck is (can be a NIC, a disk, a part of network and also a mechanism of virtualization).

How did you measure the bandwidth? Recommended to use iperf3, which can be used to measure between any endpoints, e.g. from VM to host, from host to NAS etc. When using iperf3 also influences from slow disks are excluded.

 

[guletz]

VMbr02 which has one NIC that I'm using as a poin

I guess that you a have a bridge who has 2 slave interface: one from the server host and the ppp interface to the nas. In this case check if both slaves has the same MTU 9000.

 

[manf0001]

As a first step it's recommended to make some tests in order to figure out where the bottleneck is (can be a NIC, a disk, a part of network and also a mechanism of virtualization).

How did you measure the bandwidth? Recommended to use iperf3, which can be used to measure between any endpoints, e.g. from VM to host, from host to NAS etc. When using iperf3 also influences from slow disks are excluded.

Here is a screen shot of my two Iperf the top one is from the host to my Qnap NAS, and the second one with the lower speeds, is my Cent OS 7 Guest.

I have double checked my MTU settings. The Bridge for my .9 network has a MTU of 9000, my guest VM has an MTU of 9000 and the Ethernet port used on the QNAP for the link between the two has an MTU of 9000.

 

[Richard]

Here is a screen shot of my two Iperf the top one is from the host to my Qnap NAS, and the second one with the lower speeds, is my Cent OS 7 Guest.

And how looks iperf3 result from guest 192.168.9.5 to host 192.168.9.9 like? Check it in order to be sure to identify the guest-host connection as bottleneck.

 

[manf0001]

And how looks iperf3 result from guest 192.168.9.5 to host 192.168.9.9 like? Check it in order to be sure to identify the guest-host connection as bottleneck.

This is the output when I do Gust to Host, the second picture is Host to Guest.

 

[Richard]

This is the output when I do Gust to Host, the second picture is Host to Guest.

Shows that the virtual NIC is not the bottleneck. Probably something wrong with MTU settings.

What I would do next: create in the same VM an additional virtual NIC (with standard MTU settings) and verify if connection to another endpoint at the physical LAN works well (i.e. bandwidth ~ 1Gb/s).

 

[manf0001]

Shows that the virtual NIC is not the bottleneck. Probably something wrong with MTU settings.

What I would do next: create in the same VM an additional virtual NIC (with standard MTU settings) and verify if connection to another endpoint at the physical LAN works well (i.e. bandwidth ~ 1Gb/s).

Ok so I tested the new Virtual NIC with automatic Settings, to my laptop hardwired in, and did the iperf test, and it shows 1.09Gbytes Transfer and 936MBits/sec Bandwidth on both the host and guest.

So I tried from the guest to my NAS, adjusted the MTU on the NAS to 1500 and plugged it back in to each other. For some reason I couldn't ping between the two so I rebooted both and ran then ran the iperf from my guest to the NAS and it shows as 1.10GBytes Transfer and 944Mbits Bandwidth so I would think it's better but when I try to pull data down from the nas, via SMB share I'm only getting 9.3MB/sec.

So I tried copying via FTp and they speeds have dropped down to about the same speeds.. for both attempts, it starts out at 50MB/sec then just drops.

 

[Richard]

For some reason I couldn't ping between the two so I rebooted both and ran then ran the iperf from my guest to the NAS and it shows as 1.10GBytes Transfer and 944Mbits Bandwidth so I would think it's better but when I try to pull data down from the nas, via SMB share I'm only getting 9.3MB/sec.

So I tried copying via FTp and they speeds have dropped down to about the same speeds.. for both attempts, it starts out at 50MB/sec then just drops.

Filetransfer performance (regardless if done via (s)ftp, nfs or samba) does not depend on network speed only but also from CPU load, disk access and more.