PVE GUI Reporting higher CPU usage than VM is reporting

[Tmanok]

Hi Everyone,

Running a standard Debian VM with a webserver and qemu-guest-agent, I'm wondering why PVE sees ~6-8% CPU usage when the VM is idle and the VM itself internally sees 0.1-1.8% while idle. Here are two screenshots showing PVE GUI and VM Top or htop at the same time to better illustrate this:




I've left the system alone for a long period of time and redirected all traffic away from it. I've also rebooted and checked other processes. What caught my eye first was a failing service, previously PVE was showing 15% usage (alarmed me because this VM was supposed to be idle) while the VM was showing somewhere around 4-8% usage with the failing service. So there is quite a disparity here, and it has persisted even after I removed all the CPU load that is visible to me.

The RAM usage also seems off (even while counting the buffers and caches highlighted by htop). free -mh reports 110MiB used, 10MiB shared, and 197MiB in buffers/caches, which is rather perfectly inline with htop. That accounts for ~317MiB of memory usage, so why is PVE showing 1,187.84MiB usage? (1.16GiB).

Thanks everyone, cluster version is 7.0-10, and although I have not noticed many other issues like this, it is troubling to think that there could be some incorrect reporting on my systems.

 

[Tmanok]

Quick update:
Earlier I was issuing a "reboot" command from within the VM, this time I issued a "shutdown now" and when it came back up (thanks HA), the memory reporting was accurate.

However, the CPU is still reporting 5-8% usage when internally the debian guest begs to differ at 0.3%.
Thanks,

 

[VictorSTS]

What does top in the Proxmox host shows for that VM?

top -p $(pgrep -f "kvm -id (VMID)")

In my experience, you will always see that extra bit of CPU usage related to KVM overhead due to interrupt handling and process scheduling. The overhead will be slightly bigger the more sockets/cores are set up for the VM, but I've never seen more than 8% in normal usage.

Anyway, that overhead wont take performance out of your VM, but from the host itself: try to run a 1 socket/2 core VM, install stress-ng and run it inside the VM with just 1 thread:

stress-ng -c 1

You will see 100% user usage inside the VM, but in the Proxmox host you will get something like 102% to 108% for that kvm process, meaning that the overhead is handled by another core/cpu of the host, allowing the processes inside the VM to fully get all configured CPUs.

 

[Tmanok]

Wise words! These thoughts had slipped my mind.

Unfortunately watching the resource consumption of the corresponding PID for the VM shows ~13.8-15% CPU usage (Or roughly the original 5-8% if you divide by 2 for the number of cores allocated to the VM).

Additionally I'm still abut 200MiB of memory over. free -mh shows about ~300MiB consumption inside the vm, PVE GUI displays 573MiB. I'll run a stress test to see whether the memory and CPU are still inaccurate.
Cheers,


Tmanok