Extremly High Load - proxprox

[ahoyer2]

The virtual server that is running proxmox 1.6 is being hammered and I don't know where to start to bring the load down to normal levels. Our load average was around 1-2, with 384Mb or memory and 1 processor.

Today our users were complaining that they were not getting their e-mails from outside, and that when they did, it was 2 hours later.


I checked the machine, and the load average has been at 15 or higher. The mail queue is huge (1500+ messages). I gave the virtual machine 1Gb of memory fearing that it was thrashing the swap and gave it access to 2 processors, but the load remained the same and nothing changed (other than it's using now much more memory).

If I hit the flush queue option in the queue, the numbers reset and start queueing up again. Eventually I see errors from proxprox saying that it can't connect to postfix.

I see a bunch of proxprox processes with top (at times the whole top screen has these processes only).

Here's a sample line from top
17319 root 16 0 74024 52m 2776 R 27.3 5.2 0:07.31 proxprox

I changed the order of the dns servers in resolv.conf to see if it was dns latency that was causing this, but nothing changed.

Is there anything I can do to start troubleshooting the high load?

Thanks,

Alex.

 

[ahoyer2]

Here's an error I'm getting in the logs (syslog) which I'm not sure is related but could help.

Nov 22 17:11:21 localhost /usr/bin/proxprox[24421]: bayes: cannot open bayes databases /root/.spamassassin/bayes_* R/W: lock failed: File exists

spam:~# ls -la /root/.spamassassin/bayes_*
-rw------- 1 root root 46200 Nov 22 17:12 /root/.spamassassin/bayes_journal
-rw------- 1 root root 20508672 Nov 22 17:12 /root/.spamassassin/bayes_seen
-rw------- 1 root root 41885696 Nov 22 17:12 /root/.spamassassin/bayes_toks

 

[tom]

hi,
is it possible to give us SSH access to this machine?
please sent this to t.huber@proxmox.com, our IP is 213.129.239.114.
thanks,

 

[dietmar]

Nov 22 17:11:21 localhost /usr/bin/proxprox[24421]: bayes: cannot open bayes databases /root/.spamassassin/bayes_* R/W: lock failed: File exists

This is usually no problem

- Dietmar

 

[dietmar]

Is there anything I can do to start troubleshooting the high load?

The easiest way to put the load off your server ist to stop incoming mail, for example by blocking SMTP traffic on your firewall.

Flushing the queue is a very bad idea in that situation, because it re-queues all mails man makes the load even higher.

How many mails does that server receive per day? Did you enable OCR?

- Dietmar

 

[ahoyer2]

Hi Dietmar,

It seems the machine was subjected to a spam storm. The loads have now come down to normal again.

The machine received 30000+ e-mails that day which is what it usually receives. What I did notice was that the amount of spam caught jumped from 11% (excluding grey listing) to 30%.

I did enable the OCR detection.

I'll take notice of not flushing the queue under heavy load.

Thanks,

Alex.

 

[dietmar]

I did enable the OCR detection.

OCR need high performance, and is not suitable for machine with 348Mb RAM. I suggest to disable OCR on such machines.

- Dietmar