High CPU use (18+%) by idle KVMs

[shubes]

I've noticed a much higher CPU use by some KVMs since upgrading from PVE2.1 to PVE2.2 (and PVE2.3). I suspect that the problem might be related to qemu (1.1.1 on PVE2.1, 1.2.0 on PVE2.2, and 1.4.0 on PVE2.3).

KVMs running IPCop (2.6.32 kernel) essentially idle on PVE2.1 use ~3.0% of 1 CPU. A pretty much identical KVM on PVE2.2 uses ~17% of 1 CPU. This has been observed across 3 PVE2.1 hosts, 5 PVE2.2 hosts and 2 PVE2.3 hosts.

Win2k3 KVMs also exhibit excessive CPU use. While I wouldn't consider any Win2kX server to be very idle at any point, an "idle" Win2k3 server on PVE2.1 uses ~5.5% of 1 CPU, while other "idle" Win2k3 servers (admittedly different) on PVE2.2 typically use ~30% of 2 CPUs.

KVM guests running CentOS5.x (2.6.18 kernel) also exhibit this problem, although it has not been verified on PVE2.1. COS5.x KVMs on PVE2.2 typically use 20-30% of 1 CPU.

KVM guests running CentOS6.3 do not exhibit this problem. In PVE2.1 and PVE2.2, our COS6.3 KVMs use ~0.5% CPU when idle.

Does anyone have a guess as to why CPU use has shot up since PVE2.1? I've searched the forum and google and come up empty so far.

Thanks.

 

[tom]

does it change if you disable the "tablet for pointer"?

see Option: Use tablet for pointer

 

[shubes]

Tried that. Doesn't help. Does screw up the cursor in the console window though.

 

[macday]

Perhaps the new default ballooning enabled in 2.3 ? I noticed that myself. Also I noticed, that hte balloning service is not working on some 2003 machines. On 2k8 and 7 machines everything is ok.

 

[tom]

I don´t think that the ballooning is the reason (btw, you can disable it by adding "balloon: 0" in the VMID.conf file).

post the VM config:

qm config VMID

 

[macday]

Hi Tom,

here is the config of a non-working-balloon windows 2003 machine:

acpi: 1
boot: cdn
bootdisk: scsi0
cores: 2
cpu: host
cpuunits: 2000
description: Datev Datenbankserver%0A10.11.12.57%0A
freeze: 0
ide2: none,media=cdrom
kvm: 1
memory: 4096
name: ST-DB-01
net0: virtio=0E:5C:26:11:BE:7F,bridge=vmbr10
onboot: 0
ostype: w2k3
scsi0: stor03-vm02:vm-708-disk-1,size=20488M
scsi1: stor03-vm02:vm-708-disk-2,size=20488M
scsihw: virtio-scsi-pci
sockets: 1

 

[shubes]

This KVM is running on 2.1, and uses 2.1% (ironic) of 1 CPU when essentially idle:
boot: cd
bootdisk: ide0
cores: 1
ide0: local:100/vm-100-disk-1.raw,size=1G
ide2: isos:iso/ipcop-2.0.3-install-cd.i486.iso,media=cdrom,size=61768K
memory: 128
name: tacs-net
net0: e1000=BE:64:EE:1F:1FA,bridge=vmbr0
net1: e1000=2E:68:CF:20:4F:B0,bridge=vmbr1
net2: e1000=2A:C61:07:84:3D,bridge=vmbr2
net3: e1000=E2:6D:BB:BF:9B:7D,bridge=vmbr3
onboot: 1
ostype: l26
sockets: 1
startup: order=10
tablet: 0

The tablet: 0 makes no apparent difference.

This KVM (pretty much identical) runs on 2.2, and uses 17% of 1 CPU when essentially idle:
boot: c
bootdisk: ide0
cores: 1
ide0: vms:102/vm-102-disk-1.raw,size=1G
ide2: isos:iso/ipcop-2.0.3-install-cd.i486.iso,media=cdrom,size=61768K
memory: 512
name: tacs-net
net0: e1000=DA:0A:F9:FB:7F:B5,bridge=vmbr0
net1: rtl8139=86:A0:BA:C0:20:8F,bridge=vmbr1
net2: e1000=CE:19:A5:28:97:1B,bridge=vmbr2
net3: e1000=EAB:CE:7F:8E:7F,bridge=vmbr3
onboot: 1
ostype: l26
sockets: 1
startup: order=5

Hardware is identical.

Both KVMs are running this kernel:
GNU/Linux 2.6.32-6
#1 SMP Tue Feb 14 11:01:48 CET 2012
i686 QEMU Virtual CPU version 1.2.0 AuthenticAMD

Also, debian 6 (2.6.32) exhibits this problem, while debian 7 (3.2) does not.

It's odd to me that CentOS6.3 (2.6.32) does not exhibit this problem. I wonder if RHEL backported something into their 2.6.32 kernel which alleviates the problem.

Also, a colleague of mine determined that none of these 2.6.32 KVMs exhibit this problem when running on a current Arch host (3.7.1 kernel) with qemu+kvm. FWIW.

Thanks.

 

[tom]

install a 3.2 kernel in your debian 6 hosts (from squeeze-backports).

 

[shubes]

I don't actually have any debian 6 hosts that are being used. Just did that for a test.

What about other 2.6.32 kernels, and Windows 2k3/2k8 servers?

 

[dietmar]

does it help if you use e1000 instead of rtl8139 (net1)?

 

[shubes]

does it help if you use e1000 instead of rtl8139 (net1)?

Nope. I have e1000 across the board on other KVMs which exhibit the problem.

 

[spirit]

Nope. I have e1000 across the board on other KVMs which exhibit the problem.

update your kernel to 3.2 if you can, I have see same behaviour with 2.6.32 kernel on debian.

 

[shubes]

Upgrading to a 3.2 kernel in the KVMs will likely fix things all right. Problem is, that's not an option in some cases. I'm wondering if there's something I can backport from 3.2 which will fix up 2.6.32 kernels appropriately. RHEL has apparently done this, as COS6 (2.6.32) doesn't have this problem.

I also expect that upgrading the PVE host kernel to 3.2 (or perhaps later) will fix things for all KVMs, since a current Arch with qemu+kvm doesn't have this problem.

 

[mo_]

the problem is that openvz strictly uses RHEL kernels and only the new RHEL 7, arriving in the second half of 2013 will have a 3.8 kernel

 

[shubes]

Not exactly *the* problem here. The problem here is strictly with KVMs, not openvz containers.

BTW, I'm also looking for performance improvements for Windoze KVMs. I have a few win2k3 KVMs that are eating 30% of 2 CPUs just idling (if there is such a thing for Windows machines). Win7 looks better though (only 10%).

Thanks.

 

[mo_]

I also expect that upgrading the PVE host kernel to 3.2 (or perhaps later) will fix things

this is what you said. the proxmox kernel uses KVM and openvz modules meaning there cant be a 3.8 pve kernel until rhel7 is released.

 

[spirit]

we have made a bugreport on qemu mailing some month ago, if you want more informations

[h=1]https://lists.gnu.org/archive/html/qemu-devel/2012-11/msg00579.html[/h]

 

[shubes]

this is what you said. the proxmox kernel uses KVM and openvz modules meaning there cant be a 3.8 pve kernel until rhel7 is released.

I don't follow the reasoning here. PVE is based on Debian, not RHEL.

 

[tom]

I don't follow the reasoning here. PVE is based on Debian, not RHEL.

Proxmox VE is based on Debian AND uses a RHEL based Kernel (due to OpenVZ)

 

[shubes]

Proxmox VE is based on Debian AND uses a RHEL based Kernel (due to OpenVZ)

I see (said the blind man)!
Thanks Tom.