Stability of kvm guests

[nextime]

Hello all.
I'm testing proxmox ve for server consolidation.

I have a quad-core AMD Opteron @ 2.2 Ghz, with 8 gigs of RAM and 2x1TB sata disks with raid 1 HW, on which there is a clean debian lenny base system installed with packages (kernel comprised) from ftp://download.proxmox.com/debian repository, updated to latest available packages.

I have installed 3 identical guests, with this config:

- 120 GB qcow disks configured as virtio
- Ethernet as virtio
- 1000 cpu cycle ( but tryed also with 10000 )
- 1024 MB of ram

Inside of those vm, i have installed:

- OS: Linux Debian (testing) from netinstall (base system only)
- Partitioning:
* vda1 -> 100 MB clean ext3 mounted as /boot
* vda2 -> 2048 MB dm-crypt AES256 with random key used for swap
* vda3 -> all the remaining space with dm-crypt AES256 on which i have a VolumeGroup LVM2, divided in 2 LogicalVolume, one of 20 GB and one of the rest of the space available, both in ext3

Apparently all work well, but i have something the "sensation" than the guest systems are something a little slow, expecially during intensive IO.

With clean install, i was trying to produce an iso installer with my specific configs, so, i try to use mondorescue for that.

Mondorescue can produce an "hot" live system installable iso backup image, and, of course, it make intensive I/O during the procedure to produce the iso images.

When doing that, the system goes very high load, is unresponsive ( i can't do nothing at all for many minutes ), ram is full, swap is empty, and i get a lot of kernel debugs in the kern.log, syslog and dmesg log files like the one attached here.

Any suggestion?

 

[tom]

Hello all.
I'm testing proxmox ve for server consolidation.

I have a quad-core AMD Opteron @ 2.2 Ghz, with 8 gigs of RAM and 2x1TB sata disks with raid 1 HW, on which there is a clean debian lenny base system installed with packages (kernel comprised) from ftp://download.proxmox.com/debian repository, updated to latest available packages.

I have installed 3 identical guests, with this config:

- 120 GB qcow disks configured as virtio
- Ethernet as virtio
- 1000 cpu cycle ( but tryed also with 10000 )
- 1024 MB of ram

Inside of those vm, i have installed:

- OS: Linux Debian (testing) from netinstall (base system only)
- Partitioning:
* vda1 -> 100 MB clean ext3 mounted as /boot
* vda2 -> 2048 MB dm-crypt AES256 with random key used for swap
* vda3 -> all the remaining space with dm-crypt AES256 on which i have a VolumeGroup LVM2, divided in 2 LogicalVolume, one of 20 GB and one of the rest of the space available, both in ext3

Apparently all work well, but i have something the "sensation" than the guest systems are something a little slow, expecially during intensive IO.

With clean install, i was trying to produce an iso installer with my specific configs, so, i try to use mondorescue for that.

Mondorescue can produce an "hot" live system installable iso backup image, and, of course, it make intensive I/O during the procedure to produce the iso images.

When doing that, the system goes very high load, is unresponsive ( i can't do nothing at all for many minutes ), ram is full, swap is empty, and i get a lot of kernel debugs in the kern.log, syslog and dmesg log files like the one attached here.

Any suggestion?

first, check if you run the latest version. second, can you test this without virtio? (block and net) - any difference? and also without encryption? and try raw instead of qcow2 disk images.

 

[nextime]

first, check if you run the latest version. second, can you test this without virtio? (block and net) - any difference? and also without encryption? and try raw instead of qcow2 disk images.

Yes, it is the latest version available by apt repository (download.proxmox.com).

Now i'm doing some tests, first of all i'm trying to upgrade the guest kernel from 2.6.26-2 (standard debian from apt) to 2.6.30 (upgrading guest to sid, debian standard kernel ) to see if more updated virtion drivers can do the tricks.

I'm also trying to use a more update kernel in the host, 2.6.27 vanilla kernel + openvz patch using the config from your 2.6.24-7, but at the first try i have some problem to boot it that i need to debug.

if those things will not work, next i will try to change the file backend to raw disk images and, if also this won't fix, i'll try to remove also the encryption layer. Anyway, for those two things, i'm using qcow just couse it is the default ( and only ) option available from the web gui actually, and i'm trying to use a system where i can no touch anything on the host but a web gui. For encryption, it is a must for what i need to do sadly, so, if the problem is encryption, will be a huge problem for me.

Thanks anyway for your suggestion, i will let you all know the results of my tests

 

[tom]

Yes, it is the latest version available by apt repository (download.proxmox.com).

Now i'm doing some tests, first of all i'm trying to upgrade the guest kernel from 2.6.26-2 (standard debian from apt) to 2.6.30 (upgrading guest to sid, debian standard kernel ) to see if more updated virtion drivers can do the tricks.

I'm also trying to use a more update kernel in the host, 2.6.27 vanilla kernel + openvz patch using the config from your 2.6.24-7, but at the first try i have some problem to boot it that i need to debug.

if those things will not work, next i will try to change the file backend to raw disk images and, if also this won't fix, i'll try to remove also the encryption layer. Anyway, for those two things, i'm using qcow just couse it is the default ( and only ) option available from the web gui actually, and i'm trying to use a system where i can no touch anything on the host but a web gui. For encryption, it is a must for what i need to do sadly, so, if the problem is encryption, will be a huge problem for me.

Thanks anyway for your suggestion, i will let you all know the results of my tests

The upcoming new GUI will allow much more option for storage, please post your results and do not forget to include what Kernel do you use (Proxmox Kernel or other, pls specifiy).

 

[nextime]

Re: Stability of kvm guests - UPDATE

Little update:

- HOST kernel: default latest proxmox kernel from apt
- HOST distro: standard debian lenny with pve packages installed from apt
- GUEST distro: standard clean base install of debian lenny, with ext3 fs without nor dmcrypt or lvm, 2 partitions, 1 of 20 gigs with base system, second partition non mounted
- KVM config: qcow file disk, 1 GB of ram

Trying to run shred on the second partition to prepare it for dmcrypt, same problem as above messages.

So, now we can exclude that the problem is on the guest lvm and/or dmcrypt.

 

[nextime]

Re: Stability of kvm guests - PROBLEM FINDED

Ok, my last try was to use:

- HOST kernel: standard 2.6.24-7 pve-kernel package
- HOST distro: standard debian lenny with pve packages
- GUEST kernel: 2.6.26 and 2.6.30 both from debian packages
- GUEST disto: debian sid

HDD and ETH both with virtio, but using raw disk image instead of qcow completely solve the problem. The guest is a LOT more responsive, and even during intensive cpu load and I/O on disk, it work great and it appears to be responsive.

So, the problem is related to the use of qcow file. I just can't wait for the next release where we can use the new annunced storage poll...

 

[tom]

Re: Stability of kvm guests - PROBLEM FINDED

Ok, my last try was to use:

- HOST kernel: standard 2.6.24-7 pve-kernel package
- HOST distro: standard debian lenny with pve packages
- GUEST kernel: 2.6.26 and 2.6.30 both from debian packages
- GUEST disto: debian sid

HDD and ETH both with virtio, but using raw disk image instead of qcow completely solve the problem. The guest is a LOT more responsive, and even during intensive cpu load and I/O on disk, it work great and it appears to be responsive.

So, the problem is related to the use of qcow file. I just can't wait for the next release where we can use the new annunced storage poll...

thanks for update - we are working hard to prepare the first beta for August.