[SOLVED] CPU of kvm take 100% extra cpu time after a while

[marigo]

Hi all,

I run a kvm with some surveillance software that processes 6 streams of IP camera's. When the kvm is booted it takes about 10% of CPU. After a while, it can take 5 days or sometimes 2 weeks the CPU takes double the amount of the initial boot and runs on 20% of CPU capacity.

I thought is was some hardware issue, so I build a new server on a i5 9400, but just this kvm has the same issue, so it's not hardware related.

Somehow there is a trigger that will consume double the amount of CPU cycles, but I don't know where to look. Does someone have any tip where to look at when this is happening?
I have tried to select a different cpu type, from kvm64 to 'host', but that didn't do the trick. When I reboot the kvm it will run for a few days normally.

My hardware

 

[aaron]

Have you monitored the CPU usage inside the VM? That might tell you what is consuming so much.

 

[marigo]

I had in the past but to be honest I don't know exactly which process took more cpu power, but I thought it was the camera application.

I have written down the current cpu statistics and will compare them when the cpu load is higher.

Thank you so far.

 

[marigo]

I think I have found something what is eating CPU cycles.
The camera application is using "CPU time" to run on the VM. This counter is increasing by some time, but will not decreasing for some reason. I think this VM is getting too short time from the fair mechanism of the Proxmox VE engine. When the "CPU time" hits about 20 hours it suddenly increases the CPU load on the KVM engine.

What I did is increased the CPU Units to 10240 to give the VM 5 times the amount of CPU resources then other running VM's on the host.
The "CPU time" still increases, but it goes somehow a bit slower.

First picture everything is still ok, but second picture the CPU load is increased.






EDIT: Above "CPU time, has nothing to do with this issue"

 

[marigo]

Adding more cpu units did not help, but I think I have found the root cause of this behaviour from dmesg:

[ 6011.054983] clocksource: timekeeping watchdog: Marking clocksource 'tsc' as unstable because the skew is too large:
[ 6011.054983] clocksource:                       'hpet' wd_now: f761685f wd_last: f25e380d mask: ffffffff
[ 6011.054983] clocksource:                       'tsc' cs_now: 141f0c7a428e7 cs_last: 141f071183d45 mask: ffffffffffffffff
[ 6011.393995] clocksource: Switched to clocksource hpet

This is only seen on one kvm guest. No other vm's have this issue.
What can I do about this? The clocksource tsc eats less cpu resources then hpet.

I know that, for somehow, the clock source 'tsc' is being inaccurate. But can I tweak some setting for the vm to stay on 'tsc' as the clock source?
The NTP clock is synchronised on both the host and vm, but I think it has nothing to do with 'tsc'.

---------------------------------------------------------------------------------------------------

12-1-2020 - UPDATE1: I have applied "tsc=reliable" to the GRUB boot parameters of the guest. Hopefully this will help. The guest is running for 2 days and it's stable.
20-1-2020 - UPDATE2: Still running stable for 10 days
27-1-2020 - UPDATE3: Still stable for 17 days
03-2-2020 - UPDATE4: Still stable for 23 days

I'll guess this was the solution: add "tsc=reliable" at grub boot parameters.
To check if the guest is loaded with this statement check on guest VM:

cat /proc/cmdline

source: https://forum.manjaro.org/t/grub-cmdline-parameters/105379/3