VM CPU usage increased after proxmox upgrade from pve6.4 to pve7.0

[ferrets]

I've upgraded my proxmox ve from pve6.4 to pve7.0 last night and I found that the CPU usage of VMs had increased around 20%~40%, and I've total no idea what's the reason.

 

[Bunnikins]

I thought I was going mad - seen exactly the same thing - my node cpu load and % cpu usage appears to have tripled on all VMs and on the node as a whole.

For reference:

CPU(s)

4 x Intel(R) Core(TM) i3-8109U CPU @ 3.00GHz (1 Socket)

Kernel Version

Linux 5.11.22-1-pve #1 SMP PVE 5.11.22-2 (Fri, 02 Jul 2021 16:22:45 +0200)

PVE Manager Version

pve-manager/7.0-8/b1dbf562

Repository Status

Proxmox VE updates Production-ready Enterprise repository enabled

 

[birdy]

Saw this already in 6.4 after changing kernel to 5.11.

https://forum.proxmox.com/threads/kernel-5-11.86225/page-5#post-400408

 

[birdy]

https://forum.proxmox.com/threads/proxmox-ve-7-0-released.92007/page-6#post-402309

 

[ferrets]

https://forum.proxmox.com/threads/proxmox-ve-7-0-released.92007/page-6#post-402309

oh, if that is reason, I'm totally fine with it. I like lower power consumption( only if the max performance is the same as before).

 

[Bunnikins]

Sounds promising, if that's the case I'm cool with that.

 

[chrcoluk]

im guessing the scheduler isnt tuned well to respond to moderate levels of load.

my local proxmox currently is plugged into a watt meter, so I may compare the schedulers to see actual effect, but I already have 30w saving over esxi at idle and its on the performance scheduler 55w vs 92w.

 

[nullify]

I have this same issue after recently upgrading from PVE6.4 to PVE7.1-7. I'm using a Xeon E-2276G.

Windows VMs are very sluggish and CPU Load is constantly high, despite scaling governor being set to "performance".
Is there a workaround for this?

 

[Ymac]

I resolved it by disabling memory integrity in windows security, is it enabled in your case?
https://forum.proxmox.com/threads/windows-guest-slow-after-pve-6-to-7-upgrade.93302/#post-435442

 

[Jorge Teixeira]

I have the same problem since yesterday after updating to latest kernel and Qemu version. The memory integrity was already disabled and the problem is the same.

 

[Jorge Teixeira]

On a windows server 2019 vm the cpu load was on average 15% and now it is more than 60%. This machine was already on proxmox 7.1 and working fine. Yesterday i updated it with latest kernel and qemu. The scaling governor is set to perfromance also.

 

[Jorge Teixeira]

Booting with 5.11 kernel seems to solve the problem. Will report back after some more testing.

 

[Jorge Teixeira]

After booting 5.11 kernel the problem disappeared. In the images you can see de diference.

 

[nullify]

I resolved it by disabling memory integrity in windows security, is it enabled in your case?
https://forum.proxmox.com/threads/windows-guest-slow-after-pve-6-to-7-upgrade.93302/#post-435442

Just like @Jorge Teixeira, the memory integrity setting was already disabled for me.

Rolling back to Kernel 5.11 seems to help a little, but it's still much slower than in earlier versions. It's somewhat erratic, some tasks work fine others are unbearably slow.

 

[nullify]

Okay, good news. I just updated to the recently released pve-kernel-5.15.

It completely fixed the issue for me, CPU load seems to have dropped by 50-75%. Seems like the scheduler changes that were introduced in 5.11 had some bugs.

It's a shame that PVE 7.1 released with 5.13, but you can get 5.15 from the standard repositories.
https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/

 

[Jorge Teixeira]

I also instaled the 5.15 kernel and everything looks ok. Will try it in the coming days.