Slow behavior

[plight]

I am getting the following when proxmox is running without a guest machine active. Why would there be an IO delay of 3.31% with no guests active?

Uptime15:45:48 up 00:20, load average: 0.28, 0.51, 0.30 CPU(s)2 x Intel(R) Xeon(R) CPU            3050  @ 2.13GHz 

CPU Utilization
[CENTER]1.61%[/CENTER]
 IO Delays[CENTER]3.31%[/CENTER]
 Physical Memory (1.96GB/141MB)[CENTER]141MB[/CENTER]
 Swap Space (4.00GB/0KB)[CENTER]0KB[/CENTER]
 HD Space root (18.21GB/1.29GB)[CENTER]7.49%[/CENTER]

 Version (package/version/build)pve-manager/1.4/4390 Kernel VersionLinux 2.6.24-8-pve #1 SMP PREEMPT Fri Oct 16 11:17:55 CEST 2009

I did notice in the log file there are a huge amount of mails each second, which I could see causing this error, but where are these mails coming from?

ct 30 15:46:56     qmgr    4346     73E866DD0E: from=<root@proxtest2.localdomain>, size=1264, nrcpt=1 (queue active)
Oct 30 15:46:57     qmgr    4346     7FA7B6DA6D: from=<root@proxtest2.localdomain>, size=1264, nrcpt=1 (queue active)
Oct 30 15:46:57     error    4687     7D91F6D722: to=<test@test.com>, orig_to=<root>, relay=none, delay=365816, delays=365815/0.13/0/0.11, dsn=5.0.0, status=bounced (atest.com)
Oct 30 15:46:57     cleanup    4752     271916C764: message-id=<20091030204657.271916C764@proxtest2.localdomain>
Oct 30 15:46:57     error    4746     7FA7B6DA6D: to=<test@test.com>, orig_to=<root>, relay=none, delay=239816, delays=239816/0.01/0/0.16, dsn=5.0.0, status=bounced (atest.com)
Oct 30 15:46:57     error    4779     73E866DD0E: to=<test@test.com>, orig_to=<root>, relay=none, delay=139015, delays=139015/0.06/0/0.26, dsn=5.0.0, status=bounced (atest.com)
Oct 30 15:46:57     cleanup    4744     4E2BD6C7C4: message-id=<20091030204657.4E2BD6C7C4@proxtest2.localdomain>
Oct 30 15:46:57     bounce    4749     7D91F6D722: sender non-delivery notification: 271916C764
Oct 30 15:46:57     cleanup    4684     6CEA86C7CA: message-id=<20091030204657.6CEA86C7CA@proxtest2.localdomain>
Oct 30 15:46:57     qmgr    4346     271916C764: from=<>, size=3057, nrcpt=1 (queue active)
Oct 30 15:46:57     bounce    4689     7FA7B6DA6D: sender non-delivery notification: 4E2BD6C7C4
Oct 30 15:46:57     local    4751     271916C764: to=<root@proxtest2.localdomain>, relay=local, delay=0.77, delays=0.48/0/0/0.29, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")
Oct 30 15:46:57     qmgr    4346     7D91F6D722: removed
Oct 30 15:46:57     qmgr    4346     4E2BD6C7C4: from=<>, size=3057, nrcpt=1 (queue active)
Oct 30 15:46:58     qmgr    4346     6CEA86C7CA: from=<>, size=3057, nrcpt=1 (queue active)
Oct 30 15:46:58     bounce    4741     73E866DD0E: sender non-delivery notification: 6CEA86C7CA
Oct 30 15:46:58     qmgr    4346     271916C764: removed
Oct 30 15:46:58     qmgr    4346     7FA7B6DA6D: removed
Oct 30 15:46:58     local    4753     4E2BD6C7C4: to=<root@proxtest2.localdomain>, relay=local, delay=0.92, delays=0.73/0.11/0/0.08, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")

Same question as before - what exactly is it trying to send mail for if none of the VMs are active?

edit:
Here are my pveperf results:
CPU BOGOMIPS: 8537.00
REGEX/SECOND: 686414
HD SIZE: 18.21 GB (/dev/pve/root)
BUFFERED READS: 67.05 MB/sec
AVERAGE SEEK TIME: 11.08 ms
FSYNCS/SECOND: 131.29
DNS EXT: 990.59 ms
DNS INT: 1998.74 ms (alonusa.com)

 

[plight]

I traced down the emails and they were related to a config file that was in the wrong format (but was never attached to a vm, so I do not know why it was griping unless it parses all config files regardless if they are in use or not)

I then flushed the postfix mail queue because I was receiving bounce back messages from 2 days ago...

Now, I am getting the following:

Oct 30 16:48:18     proxwww    3369     Starting new child 3369
Oct 30 16:48:22     proxwww    3373     Starting new child 3373
Oct 30 16:49:21     proxwww    3434     Starting new child 3434
Oct 30 16:49:52     proxwww    3465     Starting new child 3465
Oct 30 16:50:01     cron    3479     (root) CMD (/usr/share/vzctl/scripts/vpsreboot)
Oct 30 16:50:01     cron    3481     (root) CMD (/usr/share/vzctl/scripts/vpsnetclean)
Oct 30 16:50:01     cron    3483     (root) CMD (test -x /usr/lib/atsar/atsa1 && /usr/lib/atsar/atsa1)
Oct 30 16:50:12     proxwww    3519     Starting new child 3519
Oct 30 16:50:34     proxwww    3541     Starting new child 3541
Oct 30 16:51:22     ntpd    3129     synchronized to 67.18.187.111, stratum 2
Oct 30 16:51:26     proxwww    3596     Starting new child 3596
Oct 30 16:51:42     proxwww    3596     update ticket
Oct 30 16:51:42     proxwww    3612     Starting new child 3612
Oct 30 16:52:27     proxwww    3653     Starting new child 3653
Oct 30 16:52:29     proxwww    3655     Starting new child 3655
Oct 30 16:53:30     proxwww    3715     Starting new child 3715
Oct 30 16:54:01     proxwww    3747     Starting new child 3747

What are all those child objects?


edit: So those are just apache processes. And are probably not the cause of my IO Delay issue. Any ideas?

 

[dietmar]

yes, 'proxwww' is just apache

 

[plight]

Went ahead and did some more research.

I ran a mysql process on the physical machine and the performed the exact same function on the VM. It took a couple hours on the physical machine and a couple DAYS(!) on the VM.

The VM has Xeon 3050 and the physical has a Xeon E5405. These processors are pretty comparable aside from the E5405 having a 6meg cache and the 3050 a 2 meg cache. ide0 and ide1 on the KVM guest are both qcow images. VM is running on 1 500gig 7200 and the physical is running on a raid 5.

Is there any known issue with mysql running slowly on a guest? Any suggestions that can possibly speed this up? I dont quite understand why right now i am showing (on average) 44% cpu load and 30% IO Delays.

 

[dietmar]

I ran a mysql process on the physical machine and the performed the exact same function on the VM. It took a couple hours on the physical machine and a couple DAYS(!) on the VM.

How can I reproduce the behaviour?