Server sometimes slow

[mangoo]

So the load happens from time to time.

So, run iotop *and* htop when you see the server is slow, and you'll find the guilty.

Running these commands at different times and pasting their output here does not make much sense.

 

[typo3usa]

really - who would have guessed
load on the server i posted that from when I posted was over 74% i/o wait
if you see the trend report there - you can see much of the time it is 50%+ for the one server still left running on the 1.4

hope that helps a bit

for example:

top - 06:02:08 up 1 day, 10:56, 1 user, load average: 5.03, 4.90, 3.03

Tasks: 365 total, 1 running, 360 sleeping, 0 stopped, 4 zombie

Cpu(s): 9.5%us, 2.5%sy, 0.1%ni, 18.4%id, 49.1%wa, 0.0%hi, 0.3%si, 0.0%st

Mem: 18485780k total, 18182700k used, 303080k free, 610724k buffers

Swap: 4194296k total, 2716k used, 4191580k free, 13822548k cached

Notice that wait right now ... its 49.1% and load is 5.03

CPU is not that bad however .....

 

[dietmar]

you could just configure mlocate to ignore different (and large) directories as well -

our sar showing:

00:00:01 cpu %usr %nice %sys %irq %softirq %wait %idle _cpu_
..

sar can report many things - see man 'sar' (simply pasting the output here does not make sense - instead, try to find out whats wrong by generating different reports with sar).

 

[typo3usa]

sar can report many things - see man 'sar' (simply pasting the output here does not make sense - instead, try to find out whats wrong by generating different reports with sar).

your right - edited it
did not realize what i posted - as its roughly 6am here - been working now into a 30+ hr nonstop work load sadly...

 

[typo3usa]

this system is running mysql
mysql appears to be the bad guy

Adaptec Raid Controller 512mb ram Raid 10

notice the wait % up as high as 100%

04:00:01 sda (8-0) 73% 96.48 15.8 179.53 27.1 16.19 2.65 ms
04:00:01 sda2 345.64 4.4 1217.29 4.0
04:10:01 sda (8-0) 48% 61.89 28.3 175.42 26.5 14.89 2.04 ms
04:10:01 sda2 376.56 4.7 1160.04 4.0
04:20:02 sda (8-0) 46% 60.57 50.5 171.12 24.9 14.40 1.99 ms
04:20:02 sda2 732.22 4.2 1065.91 4.0
04:30:01 sda (8-0) 64% 89.94 28.8 166.91 25.4 17.08 2.50 ms
04:30:01 sda2 611.45 4.2 1060.15 4.0
04:40:01 sda (8-0) 45% 68.29 18.7 155.94 27.7 13.37 2.00 ms
04:40:01 sda2 293.59 4.3 1079.36 4.0
04:50:01 sda (8-0) 18% 9.31 13.1 160.23 27.1 6.96 1.03 ms
04:50:01 sda2 10.67 11.4 1084.52 4.0
05:00:01 sda (8-0) 17% 8.72 14.2 161.72 25.5 5.46 0.97 ms
05:00:01 sda2 10.62 11.7 1030.00 4.0
05:10:01 sda (8-0) 17% 8.28 20.7 160.45 25.3 7.73 1.02 ms
05:10:01 sda2 9.81 17.4 1013.56 4.0
05:20:01 sda (8-0) 23% 12.34 24.6 188.17 26.9 9.10 1.13 ms
05:20:01 sda2 15.93 19.0 1263.61 4.0
05:30:01 sda (8-0) 15% 7.18 14.3 159.01 22.9 5.85 0.93 ms
05:30:01 sda2 8.34 12.3 910.87 4.0
05:40:01 sda (8-0) 22% 11.57 9.0 170.11 24.1 7.69 1.19 ms
05:40:01 sda2 12.57 8.3 1025.93 4.0
05:50:01 sda (8-0) 18% 8.25 16.0 165.02 25.2 7.01 1.04 ms
05:50:01 sda2 9.57 13.8 1040.68 4.0
06:00:01 sda (8-0) 58% 127.42 91.8 161.66 28.5 12.94 2.01 ms
06:00:01 sda2 172.65 67.8 1150.51 4.0
06:10:01 sda (8-0) 99% 245.13 5.7 159.30 30.0 8.52 2.45 ms
06:10:01 sda2 252.43 5.5 1192.80 4.0

 

[dietmar]

this system is running mysql
mysql appears to be the bad guy

Usually, an database does many fsyncs, so it is likely to increase io-load.

Adaptec Raid Controller 512mb ram Raid 10

Whats the output of (please run when there is no load on the server)

# pveperf

 

[typo3usa]

Usually, an database does many fsyncs, so it is likely to increase io-load.



Whats the output of (please run when there is no load on the server)

# pveperf

right -

with load:

pveperf
CPU BOGOMIPS: 15963.19
REGEX/SECOND: 540674
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 58.46 MB/sec
AVERAGE SEEK TIME: 45.07 ms
FSYNCS/SECOND: 93.94
DNS EXT: 250.86 ms

without load (shutdown all the vps's)

pveperf
CPU BOGOMIPS: 15963.19
REGEX/SECOND: 550932
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 232.89 MB/sec
AVERAGE SEEK TIME: 7.88 ms
FSYNCS/SECOND: 2630.51
DNS EXT: 65.98 ms

without load top showing:

top - 06:40:36 up 1 day, 11:34, 1 user, load average: 0.34, 4.27, 5.04

Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 18485780k total, 385636k used, 18100144k free, 56548k buffers
Swap: 4194296k total, 2664k used, 4191632k free, 13548k cached

im happy i can post and hopefully help this get figured out for the community

 

[hisaltesse]

I am also running Adaptec Raid Controller 512mb ram Raid 10
with 1.4 on Intel Nehalem 5520.

I do not have an IO load issue but I notice mysql spikes time to time and the server gets unresponsive time to time. I have installed sysstat and will check it in a few hours.

 

[hisaltesse]

Dietmar
What would be a meaningful sar command that you would like me to run to show stats?

 

[dietmar]

What would be a meaningful sar command that you would like me to run to show stats?

See the post from typo3usa. But you simple need to track down the problem, until you find something.

Also, it would be quite interesting if you also answer the last questions I asked (pveperf, how many mysql transactions per second).

Do you run OpenVZ only?

 

[hisaltesse]

I only run openvz and nothing else.

I just ran pveperf twice (20 secs apart) under current load and got the following:

pveperf
CPU BOGOMIPS: 72537.23
REGEX/SECOND: 629323
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 56.66 MB/sec
AVERAGE SEEK TIME: 36.72 ms
FSYNCS/SECOND: 159.39
DNS EXT: 125.84 ms
DNS INT: 7.18 ms

pveperf
CPU BOGOMIPS: 72537.23
REGEX/SECOND: 538386
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 355.59 MB/sec
AVERAGE SEEK TIME: 19.80 ms
FSYNCS/SECOND: 99.49
DNS EXT: 116.37 ms
DNS INT: 6.94 ms

How do you want me to find out about the number of mysql transactions per second?

 

[typo3usa]

a good way of tracing mysql as well would be to use mytop

Mytop is a Unix top command clone for mysql . It is pretty good - and can give you a large amount of information.





# apt-get install mytop


Keep in mind that mytop needs the username and password to access your mysql server - if you install mytop in on the vps itself - you can add user/pass into the my.cnf file to make life easier...



But you can also pass that info while running the command For example if your username is admin and password is Hostmedic then you can start mytop as follows:



$ mytop -u admin -p Hostmedic


Keep in mind - apt-get is debian / ubuntu - you may need to use yum or rpm depending upon the OS your using in the vps.


Example of output:

MySQL on localhost (5.0.81-community) up 0+07:34:19 [14:23:45]
Queries: 15.0 qps: 0 Slow: 0.0 Se/In/Up/De(%): 12843467/00/00/00
qps now: 2 Slow qps: 0.0 Threads: 4 ( 1/ 21) 00/00/00/00
Key Efficiency: 98.2% Bps in/out: 0.0/ 1.7 Now in/out: 41.9/ 6.2k

Id User Host/IP DB Time Cmd Query or State
-- ---- ------- -- ---- --- ----------
31875 root localhost mysql 0 Query show full processlist
31827 eximstats localhost eximstats 39 Sleep
31850 firstum1_ localhost firstum1_t 47 Sleep
31843 firstum1_ localhost firstum1_t 51 Sleep

 

[dietmar]

I just ran pveperf twice (20 secs apart) under current load and got the following:

You need to run that when there i no load on the server. Else you do not get reliable results.

How do you want me to find out about the number of mysql transactions per second?

I am not a mysql expert, sorry. But you complain that the system is slow, so how do you meassure that 'slowness' if you do not know what the system is doing? How do you messure that 1.1 is faster than 1.4?

 

[typo3usa]

How do you messure that 1.1 is faster than 1.4?

In our case it is simply user experience.
We have a large amount of users hitting the server requesting websites -
Server load (as you saw on SAR ) is high

BTW - did I apologize yet about posting the junk sar into the list here -
finally got a short amount of sleep

 

[typo3usa]

btw Dietmar - judging by the times your on the forums - i think your working the same kinda hours I am

thanks for your hard work - even if those of us in the community don't always show our appreciation

 

[dietmar]

In our case it is simply user experience.
We have a large amount of users hitting the server requesting websites -
Server load (as you saw on SAR ) is high

Sure, but server 'load' and %wait' are just some artificial numbers - The question is if performance (TPS, ...) is affected too.

finally got a short amount of sleep

good to hear ;-)

 

[typo3usa]

performance when this happens is downright horrible.

So much so - that requesting a simple webpage literally takes up to 20 seconds where as before it would be instant (and thats in network )


if it would help - i can get you access to our system during an evening when user load is low - thus helping a bit

I am thinking if others can - who knows... might be able to find the bottom issue

for all I know - it could be the version of centOS we are using on the virtual system - and a particular something or other...

(of course I am sure your hands are pretty full already )

 

[hisaltesse]

You need to run that when there i no load on the server. Else you do not get reliable results.



I am not a mysql expert, sorry. But you complain that the system is slow, so how do you meassure that 'slowness' if you do not know what the system is doing? How do you messure that 1.1 is faster than 1.4?

Dietmar,

I understand your frustration but trust me it is even more frustrating for me when our VPS users call in to complain that their server is slow and want a solution right now...

I cannot run pveperf on this server at no load because it is a production server. However this is the results of the pveperf at no load when the server was being tested:
http://www.proxmox.com/forum/showpost.php?p=10761&postcount=6

The reality is that we have some other servers running 1.1 and this new server was directly deployed with proxmox1.4b2 and yesterday updated to 1.4. This is the only server on which at times even the proxmox web interface sometimes would become unresponsive for several seconds..

The question is how can we effectively find out what is going on if you have some idea. Otherwise I have been simply checking the top command at the time when a user complains about slowness. I have been able to reproduce the slowness of vps by accessing myself the users vps when they complain.

And at some point I ran top and kept observing it for a while to see the spikes and was able to see that it seems to have something to do with the mysql processes.

All our vps run centos 5.2 or 5.3 and mysql with innodb enabled and with some innodb enhanced settings.

If there is anything else that I can do let me know.

 

[dietmar]

I understand your frustration but trust me it is even more frustrating for me when our VPS users call in to complain that their server is slow and want a solution right now...

Sorry, but I am certainly not frustrated - I just try to help you to find the problem.

The question is how can we effectively find out what is going on if you have some idea.

I guess we gave you plenty of tips in this thread. You simply need to observe and try to isolate the problem (try another hardware, try another kernel version, ...). Or you find a way to reproduce the problem soemhow - then I can debug it here.

 

[dietmar]

And at some point I ran top and kept observing it for a while to see the spikes and was able to see that it seems to have something to do with the mysql processes.

Sure, but maybe there is really such high load on the server. Usually mysql produces high load when there are many transactions. Maybe you can move that VM to another host (to run isolated). Also, what was the network activities at that time - maybe there is just too much load on the network?