VMs Network speed slowing down

PhilV · Jul 8, 2016

Hello,

I'm new with Proxmox and the first tests are very encouraging.
I installed a standard OVH server with Proxmox 4.2, with no exotic conf... and 5 VMs (2 LXC and 3 KVM)

I encounter a problem with the network throughput on virtual machines :

The Host Machine have no speed pb : average 100 MB/s
When I Start a VM, the speed is about the same : between 80 and 100 MB/s.
But after few minutes, this speed decreases to less than 10 MB/s, and sommetime 1 or 2 MB/s...
And never increase... until next reboot.

This is the same for LXC or KVM VMs...

iperf tests have showeb that this troubleshooting only affect Downloads (from outside to server), but uploads work fine.

Is there a specific setting to unleash bandwith ?

Thanks in advance for helping

(And sorry for my poor english speaking).

LnxBil · Jul 8, 2016

There is not such "throttle option over time" and I'd suggest to try to debug the network traffic itself to see where tcp congestion kicks in.

PhilV · Jul 19, 2016

Hello,
So I haven't find any solution... This is really frustating.
Network traffic is almost zero cause it is a brand new installation except SSH server and Postfix (not configured).

Host :

Code:

Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat        PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      21065/sshd
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      3079/spiceproxy
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1816/master
tcp        0      0 0.0.0.0:58464           0.0.0.0:*               LISTEN      1411/rpc.statd
tcp        0      0 0.0.0.0:8006            0.0.0.0:*               LISTEN      5679/pveproxy
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      8781/dtach
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1386/rpcbind
tcp        0      0 127.0.0.1:85            0.0.0.0:*               LISTEN      5632/pvedaemon
tcp        0      0 149.202.xxx.xxx:53       0.0.0.0:*               LISTEN      1593/named
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      1593/named
tcp        0      0 149.202.xxx.xxx:22       85.68.250.0:64460       ESTABLISHED 18111/1
tcp        0    272 149.202.xxx.xxx:22       85.68.250.0:50546       ESTABLISHED 27704/3
tcp6       0      0 :::22                   :::*                    LISTEN      21065/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1816/master
tcp6       0      0 :::37791                :::*                    LISTEN      1411/rpc.statd
tcp6       0      0 :::111                  :::*                    LISTEN      1386/rpcbind
tcp6       0      0 :::53                   :::*                    LISTEN      1593/named
udp        0      0 149.202.xxx.xxx:53       0.0.0.0:*                           1593/named
udp        0      0 127.0.0.1:53            0.0.0.0:*                           1593/named
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1386/rpcbind
udp        0      0 0.0.0.0:33001           0.0.0.0:*                           12812/systemd-times
udp        0      0 0.0.0.0:703             0.0.0.0:*                           1386/rpcbind
udp        0      0 127.0.0.1:739           0.0.0.0:*                           1411/rpc.statd
udp        0      0 0.0.0.0:43280           0.0.0.0:*                           1411/rpc.statd
udp6       0      0 :::58173                :::*                                1411/rpc.statd
udp6       0      0 :::53                   :::*                                1593/named
udp6       0      0 :::111                  :::*                                1386/rpcbind
udp6       0      0 :::703                  :::*                                1386/rpcbind

And Guest :

Code:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      815/rpcbind
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1259/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1190/master
tcp        0      0 51.254.xxx.xxx:22        85.68.250.0:64457       ESTABLISHED 1945/2
tcp        0    272 51.254.xxx.xxx:22        85.68.250.0:50548       ESTABLISHED 1971/3
tcp6       0      0 :::111                  :::*                    LISTEN      815/rpcbind
tcp6       0      0 :::22                   :::*                    LISTEN      1259/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1190/master
udp        0      0 0.0.0.0:111             0.0.0.0:*                           815/rpcbind
udp        0      0 0.0.0.0:990             0.0.0.0:*                           815/rpcbind
udp6       0      0 :::111                  :::*                                815/rpcbind
udp6       0      0 :::990                  :::*                                815/rpcbind

iperf test is showing that issue is only form downloads (outside to VM) :

Code:

root@mycontainer:~# iperf -c iperf.ovh.net -i 2 -t 10 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to iperf.ovh.net, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 51.254.xxx.xxx port 42386 connected with 188.165.12.136 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 2.0 sec   217 MBytes   911 Mbits/sec
[  5]  2.0- 4.0 sec   224 MBytes   940 Mbits/sec
[  5]  4.0- 6.0 sec   224 MBytes   941 Mbits/sec
[  5]  6.0- 8.0 sec   224 MBytes   941 Mbits/sec
[  5]  8.0-10.0 sec   224 MBytes   941 Mbits/sec
[  5]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec
[  4] local 51.254.xxx.xxx port 5001 connected with 188.165.12.136 port 36293
[  4]  0.0- 2.0 sec  1.73 MBytes  7.24 Mbits/sec
[  4]  2.0- 4.0 sec  1.32 MBytes  5.53 Mbits/sec
[  4]  4.0- 6.0 sec  1.32 MBytes  5.53 Mbits/sec
[  4]  6.0- 8.0 sec  1.48 MBytes  6.21 Mbits/sec
[  4]  8.0-10.0 sec  1.12 MBytes  4.70 Mbits/sec
[  4]  0.0-10.3 sec  7.25 MBytes  5.91 Mbits/sec

Nothing in syslog...
I began to doubt the server hardware, but provider (OVH) has not confirmed and said averythings is alright.
Fresh install of Proxmox 4.2 + 1 LXC container with debian 8 template is enought to issue occurs...

So... I think I have to change server soon.

chris121212 · Jul 19, 2016

This is a limitation of OVH's connection. I got the same results on Servers under 100 Eur/month.
They say that they don't throttle - but they do. Really Painful to backup my data every day.

In the other direction 100Eur+ Server -> cheap Server is no limit. Maybe anybody can confirm that !?

LnxBil · Jul 19, 2016

If that's really the case, you should use ZFS and send/receive the incremental changes. This will really reduce the amount of backup data significantly.

LnxBil · Jul 19, 2016

Another remark:

Try to throttle the throughput yourself to approx 50% bandwidth and test if OVH is throttling after the same time or twice the time. Only to guess what they do.

PhilV · Jul 20, 2016

I'm not sure they throttle cause as I said in 1srt post, the host machine has no pb !

Code:

root@proxmox:~# iperf -c iperf.ovh.net -i 2 -t 10 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to iperf.ovh.net, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 149.202.xxx.xxx port 50298 connected with 188.165.12.136 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 2.0 sec   216 MBytes   906 Mbits/sec
[  5]  2.0- 4.0 sec   224 MBytes   941 Mbits/sec
[  5]  4.0- 6.0 sec   223 MBytes   936 Mbits/sec
[  5]  6.0- 8.0 sec   179 MBytes   752 Mbits/sec
[  5]  8.0-10.0 sec   183 MBytes   769 Mbits/sec
[  5]  0.0-10.0 sec  1.00 GBytes   860 Mbits/sec
[  4] local 149.202.xxx.xxx port 5001 connected with 188.165.12.136 port 51618
[  4]  0.0- 2.0 sec   202 MBytes   846 Mbits/sec
[  4]  2.0- 4.0 sec   224 MBytes   941 Mbits/sec
[  4]  4.0- 6.0 sec   224 MBytes   939 Mbits/sec
[  4]  6.0- 8.0 sec   224 MBytes   941 Mbits/sec
[  4]  8.0-10.0 sec   224 MBytes   941 Mbits/sec
[  4]  0.0-10.0 sec  1.08 GBytes   922 Mbits/sec
root@proxmox:~#

So, I do not know how they could throttle only the VMs...
Additionnaly, only the download is affected (outside to VM) !
I test it with many outside servers : from différents OVH Datacenters and a Online one...
The only the contractual limitation is guaranteed bandwidth at 250 Mbps (31 MB/s) for upload (OVH to Internet)

chris121212 · Jul 20, 2016

Do u have OVH, Sys or Kimi?

PhilV · Jul 20, 2016

It's a OVH HOST-64L

https://www.ovh.com/fr/serveurs_dedies/details-servers.xml?range=HOST&id=2016-HOST-64L

With 3x2 To (RAID 5) option.

For information, I have also a SoYouStart with Proxmox 4.2 and 3 VMs... and no problem, all LXC containers have full bandwidth !

icaro · Jul 21, 2016

Similar problem here, OVH-HOST-32L, main IP works fine but network IO on VM with secondary IP slows down after a short while - BUT:
I'm not even using Proxmox on this machine (yet) but the standard debian kvm/libvirtd stuff with virt-manager, I was just investigating if Proxmox could solve the issue. I tried out various kernels, standard Debian 3.16, OVH 3.14, OVH 4.4.8 - same effect, so it seems to be unrelated to that. Maybe time to open a ticket with OVH.

PhilV · Jul 21, 2016

I've opened a ticket with OVH from july/16.
They have tested on a test server "same configuration as me" as they said... with no pb.
My conf is simple : Install Proxmox, install default container (or KVM) wait 5-10 mins and download slowing...

Code:

[  4] local 51.254.xxx.xxx port 5001 connected with 188.165.12.136 port 39418
[  4]  0.0- 2.0 sec  2.62 MBytes  11.0 Mbits/sec
[  4]  2.0- 4.0 sec  1.75 MBytes  7.34 Mbits/sec
[  4]  4.0- 6.0 sec  1.91 MBytes  7.99 Mbits/sec
[  4]  6.0- 8.0 sec  1.40 MBytes  5.88 Mbits/sec
[  4]  8.0-10.0 sec  1.70 MBytes  7.11 Mbits/sec
[  4]  0.0-10.7 sec  9.88 MBytes  7.77 Mbits/sec

Unusable in production environment.

So, I suggest they try on my server... wainting for answer.
I think, I have to order another hardware... and test it.

The more time passes, more I think the problem does not come from Proxmox...

ccerrillo · Jul 29, 2016

PhilV said:
I've opened a ticket with OVH from july/16.
They have tested on a test server "same configuration as me" as they said... with no pb.
My conf is simple : Install Proxmox, install default container (or KVM) wait 5-10 mins and download slowing...

Code:

[ 4] local 51.254.xxx.xxx port 5001 connected with 188.165.12.136 port 39418 [ 4] 0.0- 2.0 sec 2.62 MBytes 11.0 Mbits/sec [ 4] 2.0- 4.0 sec 1.75 MBytes 7.34 Mbits/sec [ 4] 4.0- 6.0 sec 1.91 MBytes 7.99 Mbits/sec [ 4] 6.0- 8.0 sec 1.40 MBytes 5.88 Mbits/sec [ 4] 8.0-10.0 sec 1.70 MBytes 7.11 Mbits/sec [ 4] 0.0-10.7 sec 9.88 MBytes 7.77 Mbits/sec

Unusable in production environment.

So, I suggest they try on my server... wainting for answer.
I think, I have to order another hardware... and test it.

The more time passes, more I think the problem does not come from Proxmox...

Same issue here did you found a solution?

ccerrillo · Jul 30, 2016

@PhilV I have exactly the same issue on a server in OVH in GRA datacenter, another server in the cluster that is in RBX is not having the problem. What is your datacenter??

PhilV · Aug 1, 2016

ccerrillo said:
@PhilV I have exactly the same issue on a server in OVH in GRA datacenter, another server in the cluster that is in RBX is not having the problem. What is your datacenter??

Hi ccerrillo,
My Server is in SBG-1 datacenter...

In desperation, I've order a new server (With better CPU), install proxmox + 1 Container, and.... same results !!!!
This one is in GRA-1 datacenter

I discover that a simple wget :
from another OVH server to a Proxmox VM = OK (Full bandwith)
from any other server outside of OVH to a Proxmox VM = Very bad perf...

Code:

wget -O /dev/null http://1.testdebit.info/fichiers/200Mo.dat
--2016-08-01 14:31:58--  http://1.testdebit.info/fichiers/200Mo.dat
Résolution de 1.testdebit.info (1.testdebit.info)… 194.158.102.114, 2001:860:f70b:200::2
Connexion à 1.testdebit.info (1.testdebit.info)|194.158.102.114|:80… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 200000000 (191M)
Sauvegarde en : « /dev/null »

/dev/null                               100%[==============================================================================>] 190,73M  1,88MB/s   ds 80s

2016-08-01 14:33:18 (2,38 MB/s) — « /dev/null » sauvegardé [200000000/200000000]

I've asked to OVH support if this behavior can be due to IP Failover I use for VMs...
Waiting for answer.

ccerrillo · Aug 1, 2016

While I'm still waiting OVH support, i'm trying to find where is the problem, and i found that if you move the failover IP to a second server and then move again to the first server, the speed is normal during five to ten minutes, so the isn't a problem with the server i think the problem is in a switch

d1ch0t · Aug 9, 2016

Wish I'd noticed this thread before opening mine. Exact same issue, box in GRA. Almost 10 minutes to the second after restarting the VM's network the speed drops from 107MB/s to 3.5MB/s. If I do a systemctl restart network its back at 107MB/s. Been kicking myself for 2 days trying to figure out what I was doing/had done wrong as well....

ccerrillo · Aug 9, 2016

I'm still waiting to be solved, so I bought another server RBX. Last response i got from support was 5 days ago:

The problem seems to be somewhere between the hub SBG-4ab-a72 and routers Strasbourg SBG-4ab-a9 . The strange thing is that only affects certain servers ( Like yours ) and not all.

d1ch0t · Aug 9, 2016

Yeah, I'm not a network expert (although I've learned a lot from reading this and various other forums) but it seems like its something to do with their Virtual Mac system because if I NAT the same VM I have no issue with speed drops.

PhilV · Aug 9, 2016

Same for me,
Still waiting from support.
They ask for some Traceroute checks and said : "transfer your request also service for dysfunction Further checks"

In my case, I notice that the speed differ with the IPFO.
For example
n°1 have 15 Mbits/s
n°2 have 50 Mbits/s
And if I switch IPFO from n°2 to n°1, speed of n°1 up to 50 Mbits/s

disappointing...

amil · Aug 10, 2016

Hello,

Unfortunately the same issue here, also OVH in Gravelines datacenter.

While in dedicated server IP i get full download/upload speeds, in subnet -where i am testing the VM- the same:

- VM rebooted: i have about 8 min. the good values and the connection go very very slower and unstable.

More than one week with ticket open and no solution here: they said me that he had checked my server rack and all is okay and they suggested that i send a wget results....

Here some of my test:

In dedicated server:

./speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from OVH (37.59.15.102)...
Selecting best server based on latency...
Hosted by FreeMobile (Paris) [1.88 km]: 4.941 ms
Testing download speed........................................
Download: 825.60 Mbit/s
Testing upload speed..................................................
Upload: 310.29 Mbit/s

In a VM (subnet):

./speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from OVH (137.74.182.97)...
Selecting best server based on latency...
Hosted by FreeMobile (Paris) [1.88 km]: 5.29 ms
Testing download speed........................................
Download: 39.29 Mbit/s
Testing upload speed..................................................
Upload: 50.49 Mbit/s

And of course the machine is NOT in production, so while in node i got stable results, on subnet IPs are slow and unstable, exactly the same case.

ccerrillo said:
I'm still waiting to be solved, so I bought another server RBX. Last response i got from support was 5 days ago:

The problem seems to be somewhere between the hub SBG-4ab-a72 and routers Strasbourg SBG-4ab-a9 . The strange thing is that only affects certain servers ( Like yours ) and not all.

Did you got solved the issue? I can see that they find a hub as the root issue?

Thanks

VMs Network speed slowing down

Renowned Member

Distinguished Member

Renowned Member

Active Member

Distinguished Member

Distinguished Member

Renowned Member

Active Member

Renowned Member

New Member

Renowned Member

New Member

New Member

Renowned Member

New Member

Active Member

New Member

Active Member

Renowned Member

Active Member

We value your privacy