CPU Usage Goes to 100% After Backup Job Completes

[tman222]

Hi all,

I'm running the latest version of Proxmox VE (8.1.3) and started running into issues with my weekly backup job today. I backup 9 virtual machines to a NAS via NFS from Proxmox VE on a weekly basis. Backup mode is set to Snapshot and Zstandard is used for compression. This has been working fine for at least a couple years now. Today I noticed that the hypervisor machine's CPU usage was essentially pegged at 100% after the backup job completed (i.e. after all 9 virtual machines were backed up). Looking at individual virtual machines, their CPU usage was also maxed out. I have tried backing up just an individual VM vs. the whole set, but that doesn't appear to matter. The CPU usage of the VM still goes to 100% after the backup job completes. Changing the backup job parameters (e.g. the backup mode or compression) also doesn't seem to make a difference, and the CPU usage still goes to 100% after the backup completes. Rebooting the hypervisor machine brings the CPU usage back down to normal levels.

Does anyone have any ideas how I might begin to troubleshoot this or what to look at? Thanks in advance for your help, I really appreciate it.

 

[pama]

I do have the same issue. The virtual machines do not see that CPU usage. It's the kvm processes on the Proxmox VE hosts.

 

[tman222]

It looks like it's another instance of the issue which has already been described in this thread:

https://forum.proxmox.com/threads/pve-100-cpu-on-all-kvm-while-vms-are-idle-at-0-5-cpu.138140/

 

[lupus92]

same problem

 

[mnih]

Same problem here. Proxmox 8.1.3 cluster, only ha managed VMs seem to be affected. Restart of KVM process solves problem temporarily until next backup.

 

[Dark26]

Same problem here . High cpu for the kvm process, but nothing the vm itself.

 

[JensF]

Same problem here . High cpu for the kvm process, but nothing the vm itself.

Patch is out since Tuesday...
https://forum.proxmox.com/threads/pve-100-cpu-on-all-kvm-while-vms-are-idle-at-0-5-cpu.138140/page-2

 

[genivos]

Patch is out since Tuesday...
https://forum.proxmox.com/threads/pve-100-cpu-on-all-kvm-while-vms-are-idle-at-0-5-cpu.138140/page-2

I have this patch already installed and it doesn't help for my setup. Still goes nuts during a backup, not after .. during.

 

[Dunuin]

Restart your VM so they actually use the new qemu version?

 

[JensF]

I think a restart doesn't stop the KVM process. The VMs has to be powered down.

 

[Dunuin]

I think a restart doesn't stop the KVM process. The VMs has to be powered down.

Jup. Restart via webUI will do da proper shutdown + start so this would work. Restart from within the guestO won't work.

 

[genivos]

Jup. Restart via webUI will do da proper shutdown + start so this would work. Restart from within the guestO won't work.

The proxmox machine has been without power thrice already. It went completely unresponsive so I had to take the power off. Which means also the VMs have been powered off thrice already and been starting up. I'd say that would he enough already? But clearly didn't help since I still have the problem.

 

[kenmarcus_sd]

Possibly disable io thread option under Hardware --> Hard disk

Also, I have two identical servers, one had the issue and one did not. Most likely unrelated, but...
The one with the mixture of CTs and VMs had no issues.
The one with only VMs and one always stopped VM, did have issues.

 

[genivos]

Possibly disable io thread option under Hardware --> Hard disk

Also, I have two identical servers, one had the issue and one did not. Most likely unrelated, but...
The one with the mixture of CTs and VMs had no issues.
The one with only VMs and one always stopped VM, did have issues.

Since I have changed compression from ZSTD (Fast and good) to LZO (Fast) the problem is gone, it never happened anymore. I didn't apply any updates during the testing period, just to see how this change went. Not sure if that is or can be connected somehow to io thread though.

Also all my servers had randomly problems before, not just one.