High Cpu Load on idle KVM host

[pieteras.meyer]

Hi

Hope someone can assist me, We have just loaded Proxmox 4.2 details below. We are planning to upgrade from PVE 3.4 and create a 3 Server cluster, but we now seems to have some load issues, see below screenshot of a servers where the guest shows idle but GUI shows avg of 10% cpu usage.



# pveversion --verbose
proxmox-ve: 4.2-56 (running kernel: 4.4.13-1-pve)
pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-42
qemu-server: 4.0-83
pve-firmware: 1.1-8
libpve-common-perl: 4.0-70
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-55
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-19
pve-container: 1.0-70
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80

 

[spirit]

Hi, this could be interrupts related.
Do you have a lots of network packets going to this vm ? (As I'm seeing cumulus as vm name)

Do you see same behaviour on proxmox 3.4 ?

 

[pieteras.meyer]

Hi Spirit

This is just a test vm for now, so there is currently not any traffic going to this vm, it is totally idle.

We did not see the same behaviour on 3.4.

How can I check if it is related to interupts?

This vm was also the only one running on the system.

 

[pieteras.meyer]

Here is another example screenshot of some others servers with the same symtoms, also idle servers not doing anything.

All Debian 8.4 minimal install servers





Then there is this one Debian 8.4 minimal install only difference is it has 2 vCPU's

 

[pieteras.meyer]

HI Guys

Here is a screen shot of the same template Debian 8.4 server on our PVE 3.4 Cluster, also an Idle server

Any Ideas what the cause could be of my issues?

 

[spirit]

you can try to debug with "perf" command on host

apt-get install linux-tools-4.4

then

"perf top"

to see what is eating cpu

 

[spirit]

They are also a new feature in kvm module :"halt_poll_ns"
https://lkml.org/lkml/2015/2/6/319

This improve latency in vm, but it's increase the cpu usage of idle vm by pooling continuously the vms.

you can try to add "halt_poll_ns=0" to kvm module options

/etc/modprobe.d/kvm.conf

options kvm halt_poll_ns_grow=0 halt_poll_ns_shrink=0 halt_poll_ns=0

 

[pieteras.meyer]

Hi Spirit
Thanks you for the tip. I have done this and I think I have found my issue.

Seems it was related to standard linux bridge, I removed on lan port from the bridge and the CPU usage dropped.

 

[spirit]

Hi Spirit
Thanks you for the tip. I have done this and I think I have found my issue.

Seems it was related to standard linux bridge, I removed on lan port from the bridge and the CPU usage dropped.

View attachment 4037

So, you should check with tcpdump if you don't have broadcast or multicast flooding your brige (and vms on this bridge)

 

[pieteras.meyer]

Hi Spirit

Yes it seems that was my issue.

But now the load seems normal,

Thanks again with your help.

 

[LnxBil]

Yes, I thought about this too. Often you'll run into increased load if you do not split your cluster from your normal network. All the cluster multicast messages will be passed to each VM, therefore increased and unnecessary network load.

 

[MisterY]

I have the same problem, but i can't install linux-tools 4.4