Network Slow Machines

[Luis Cuellar]

Hi Community Proxmox

A few days ago doing some network performance tests I have a serious problem, I want to show the characteristics of my environment and how I do the test:

My Environment:

2 machines with proxmox 4.3. These have 4 physical interfaces eth0, eth1, eth2 and eth3.
I have them in Bonding mode roundrobin. A bridge vmbr0 direct with the bonding. The bridge is connected to two VMs created in my physical proxmox 2.

The physical proxmox 2 host has the IP 10.2.4.13 and the Virtual Machines has the IPs 10.2.4.2 and 10.2.4.3 with Intel e1000 card.

My other physical host proxmox 1 has IP 10.2.4.12.

The IPERF tool is used to verify the network speed in my environments. The server mode will always be in the physical proxmox 2 "IPERF -S".

--- Tests performed

1) From proxmox 1 physical to proxmox 2 physical:

Iperf -c 10.2.4.13 -t 360

41.5 GB speed in 360 seconds. NOT problem

2) From proxmox 1 physical to VM 1 proxmox 2.

39.2 GB speed in 360 seconds. NOT problem

2) From proxmox 1 physical to VM 1 and VM 2 proxmox 2 in same time.

19.8 and 19.9 GB speed in 360 seconds VM1 and VM2. NOT problem

3) From VM1 to VM2 proxmox 2.

437 GB speed VM 1 to VM2 the same proxmox in 360 seconds. Cool the speed .

4) From proxmox 1 physical to VM 1, VM 2 and proxmox 2 physical host in same time.

VM1

VM2

Proxmox Physical 2

1.41 GB VM1, 1.28 GB VM2 and 39.0 GB Proxmox Physical 2 speed in 360 seconds ins same time. . There is a big problem, since the speed in theory must distribucice equally in all the machines, in the physical as in the virtual one. I do not understand why if I generate traffic in physics, virtual machines tend to lower their traffic by 99.75%, when it should distruirce the same in 3 equal parts. . If I mount an NFS or something that generates constant traffic to a physical proxmox, it takes 99.75% of the network, and 0.25% is only for virtual machines. Before I want to clarify that I did tests with OpenVswitch and generated the same result.

If you look at the results, you will see that if you generate all the traffic only to the virtual machines your network performance is 100%, in test number 2.

I found this inconvenient because I had an NFS connected to the proxmox and with a raw virtual drive connected to a VM. When writing data to the VM everything is fine, but when writing to data in the VM and in the folder where it had mounted the raw had problems of slowness.

In several forums I found that disable the hypervisor firewall, but only increased traffic a little. Someone knows what to do to not have this type of inconvenience and that all traffic is not consumed by the physical host, thank you.

 

[manu]

Hi Luis

thanks for this detailed report

concerning 3)
I suppose you used a VirtioNIC which is a 10GB interface. Since everything is virtual you're not slowed down by the physical constraints of the real world.

4) From proxmox 1 physical to VM 1, VM 2 and proxmox 2 physical host in same time.
do you mean here VM1 and VM2 are hosted in proxmox 1 ?
I would suggest here to have a lookt at cpu consumption. Traffic over linux bridges can cost a lot of CPU since we have to packet analysys in software.

 

[Luis Cuellar]

Hi Manu, thanks for you reply.

4) The VM1 and VM2 hosted in proxmox 2 physical. The tests I do are from proxmox 1 to proxmox 2, with the virtual machines that have hosted proxmox 2.

 

[manu]

if VM1 and VM2 are hosted on proxmox2 the results you see in 4) makes sense.
930 + 30 + 33 = 993MB/s
which is the maximum thoughput of your Gigabit NIC that you measure in 1)

 

[Luis Cuellar]

Manu, as you see in the performance tests, the physical host proxmox consumes all the bandwidth in the tests, and only gives a priority of 1 GB to the VM in 360 seconds. When I test from the physical proxmox 1 to the VMs, the thoughput comes out perfect

 

[manu]

Hi Luis
Sorry I did not get what you mean before.

I think the behaviour you see is expected. If you see a peak in traffic, coming on the same physical interface, there is nothing wrong in prioritizing the host, so at least you can fix a problem on the host there in case of a DDOS for instance.

Usually people separate the management interface and the network of the VMs on two different LANs or (V)LANs so that a surge of traffic in the web gui ( uploading an ISO ) does not perturbate the VMs or the other way round.

 

[Luis Cuellar]

Hi Manu, I have 3 VLANs, one for the Proxmox administration, one for the NAS and the last one for VMs. I find this inconvenient because I mount the NAS by NFS to a .raw disk, when I see problems of speed in the network. Investigating I realized that if there is traffic in the physical proxmox, the priority of speed in the virtual machines is affected drastically. Think for the moment that Ivy OpenVswitch, but impelled the configuration with the network and Debian, and remains the same. Think back to the VLANs, and leave everything in the same network with bonding and a single bridge. But it still presents the same affectation, I do not see CPU problems I have included that I thought that only one network interface was working but discard this drawback with a tool called IPTRAF.

 

[Luis Cuellar]

HI community Promox.

Find the solution to the problem. Create two bonding interfaces in my two proxmox for testing:

eth0 and eth1 == >> bond0
eth2 and eth3 == >> bond1

The bond0 is the tag with one VLAN and the bond1 with another VLAN, each with its corresponding bridges.

As shown in the image, the IP 10.2.6.13 leaves with the bond1 and the IP 10.2.4.2 and 10.2.4.3 are the virtual machines with the bond0. This way you do not lose the speed of the network.

--The test was at the same time.
The VM1 and VM2 test speed is the 19.X GB with bond0.
The Host Proxmox 2 Physical test speed is the 40.X GB with bond1.

In the same way I see that it is not advisable to take all the interfaces of a server with a single "bond", since the physical host of proxmox if it has any traffic with either an NFS or Cluster, it will take 99.75% of the whole Traffic of the network, leaving only a single 0.25% for the virtual machines, generating immense latency for our services that go to mount. I think it is a very delicate problem and that many people are affected. You should ask yourself why there is so much latency in your virtual machines.

The million dollar question is, is it a problem directly from Linux, KVM, or Promox? Thank you for your attention.