KVM high CPU usage even without starting any VM

[hddev]

Hi,

For the last 2 weeks, the server started to undergo sudden crashes (for example : ssh not responding even though the server responds to ping), we didn't know the source of this issue because there is no log (syslog shows ^@^@^@ characters).

At first, we didn't suspect CPU usage, because it looks normal when we inspect the problem, but yesterday I spent 2 hours watching and waiting for it to crash, and when that happens it takes less than a min , CPU usage increases from 13% to 200% in a matter of seconds.

I thought the issue comes from the VM that the client uses, so I stopped it today. But even though the VM is stopped, the server keeps experiencing this issue (KVM process eats up to 200% of the CPU ).

Thank you

Best regards,

 

[dcsapak]

hi,

i'd check the logs (journal/syslog/dmesg) and e.g. top which processes consume the resources

 

[hddev]

Thank you so much.

I already did that, still no clue. Here is my screenshot. It only shows that KVM is using up to 199% of the CPU.

Thanks

 

[dcsapak]

if that left part of the screenshot is accurate, there is only 1 vm running, which would mean that is the vm using the cpu resources...

but having 2 cores out of 8 fully loaded should not slow down the host so much...

what i would check is if the storage is overloaded

 

[hddev]

Thank you .. this is the output of df -h , there is no problem with storage

root@Debian-105-buster-64-minimal ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  824K  6.3G   1% /run
/dev/md2        407G   86G  301G  23% /
tmpfs            32G   37M   32G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/md1        487M  156M  307M  34% /boot
/dev/sdb1       2.7T   50G  2.6T   2% /mnt/data
/dev/fuse        30M   20K   30M   1% /etc/pve
tmpfs           6.3G     0  6.3G   0% /run/user/0

 

[hddev]

And this is the only error I can see on dmesg, I already disabled ACPI but that did not solve the problem (So i enabled it back again), which means this is not the issue, besides, it only happens after I restart the host.

 

[hddev]

Hi,

Here is what I just did, I will check back tomorrow to see if the CPU load is stable.

Since this morning, I had to restart the server once or twice an hour because of the CPU load. I have installed some utility ( iostats, htop, ...etc) in order to better investigate .

What I found is that KVM CPU load increases even if Qemu VMs are shutdown , which may have to do with KVM config , I checked halt_poll_ns parameter :

cat /sys/module/kvm/parameters/halt_poll_ns
output : 200000

halt_poll_ns=200000 being the default.

So , I executed this command to disable this polling option :

echo "options kvm halt_poll_ns=0" | sudo tee --append /etc/modprobe.d/qemu-system-x86.conf

Then I restarted the server for the changes to take effect.

Which will make the change permanent (and create a new file qemu-system-86.conf under modprobe.d).

Now, the server is stable for about 2 hours ... I'll check back again and confirm if it is solved.

Best !

 

[hddev]

Hi,


I restarted the server yesterday (after editing halt_poll_ns=0 and cache=none), and the server seems to be stable for 20 hours now .

Best,