Ubuntu Apache KVM VM's very slow after 2 days up

[zadunn]

Ever since I updated to version 1.1 I have had the following issue:

On a two machine PVE cluster (2 identical HP DL 380's), which is basically just a deconstructed LAMP cluster, every day or two some the VM's will slow down to a crawel. I can eleminate this by migrateing the VM's to another node. I would like to figure out why this is happening. And why migrating fixes it...

The VM nodes involved look like the following:

Four KVM Ubuntu 8.10 instances installed (minimal virtual machine option selected when installing) all using the virtio drivers. **These are most affected by this issue!***

Two HAproxy nodes for LB'ing port 80 - active-passive via heartbeat (ubuntu 8.10, min virt install, virtio drivers)

Two MySQL nodes master-master replication (ubuntu 8.10, min virt install, virtio drivers)

Two MySQL Proxy nodes for LB'ing active-passive via heartbeat (ubuntu 8.10, min virt install, virtio drivers)

NFS node for apache data (ubuntu 8.10, min virt install, virtio drivers)


I have had the same issue on the MySQL nodes as well, but it is pretty consistent on the apache nodes. Syslog (PVE host or VM) doesnt show anything. At times i have seen lock ups that look like RW errors to the "hdd's" of the VMs in the console view.

Anyone else have this issue? Any suggestions on how to start trouble shooting?

Thanks,

Zach

 

[dietmar]

Try without virtio (use scsi for disks and e1000 for net) - maybe that helps?

 

[zadunn]

Stupid question, since these are internal production boxes, I try not to play too much on them, but will i need to rebuild each KVM or can i just switch the drive in the GUI and the KVM will reconfigure after reboot? Thanks,

I have also attached a console screen shot of the I/0 errors I am seeing.

 

[dietmar]

KVM or can i just switch the drive in the GUI and the KVM will reconfigure after reboot? Thanks,

It depends on the VM, because the device name change.

 

[tom]

As far as I remember on Ubuntu this should work. just stop the VM, remove the virtio disk and add this disk again as SCSI disk. (check the boot order)

But I recommend you test this on a new VM before you go for the production machines.

 

[zadunn]

Things finally came to a bit of a head today as either the NFS or one of the MYSQL VMs went into the same sort of slow down state. I went ahead and stood up two v1.0 nodes and migrated everything away from the 1.1 nodes. So far that seems to have fixed everything. I will update the thread if i see any more errors on these VMs running on 1.0.

 

[ChristianAC]

Hi

I can confirm this problem with IO-Errors.

A single VM running on a 8-core AMD-Server just pumping several
hundred MB into a mysql-DB results in the IO-errors and is than
unusable any more.

VM is ubuntu-8.04.2-server-amd64

Regards

Christian

 

[zadunn]

all my VMs are ubuntu-8.10-server-amd64. So far down grading to PVE 1.0 has fixed the issue.

 

[ChristianAC]

So you mean......

downgrade vom pve-manager V1.1-4 to pve-manager_1.0-10_all.deb

or cann you please give a dpkg -l of your pve-packages, please.

Regards

Christian

 

[zadunn]

I actually went all out and reinstalled from scratch using a proxmox 1.0 CD (mirgrating all the VMs off to another node first).

dpkg -l follows as an attachment

 

[Saahk]

Same problem here and exactly the same I/O errors :
PVE : 1.1
KVM with virtio disk and network : Ubuntu 8.04 amd64 server with eBox and Munin (Apache, PostgreSQL, PDC, etc...). Note that virtual disk is a 500GB disk.

CPU of this wm was at 180 % in PVE panel and the load average of HN was over 30,0.

No problem with a WinXP VM over KVM and virtio network, others "VM" are containers.

(Sorry for my very bad english)

 

[ChristianAC]

Hi

I just downgraded a little bit.

pve-kernel-2.6.24-1-pve 2.6.24-4
pve-kvm 75-1
qemu-server 1.0-5

With this combination I don't have any IO-probs right now (Machine is
running since 21h)

regards

Christian

 

[mangoo]

Things finally came to a bit of a head today as either the NFS or one of the MYSQL VMs went into the same sort of slow down state. I went ahead and stood up two v1.0 nodes and migrated everything away from the 1.1 nodes. So far that seems to have fixed everything. I will update the thread if i see any more errors on these VMs running on 1.0.

Most probably virtio_net was causing these problems.