VMs freeze with 100% CPU

[insightful]

Hi There,

I am facing issue with Proxmox 8.0.3.

VMs are getting freeze with 100% CPU.

Please help me regarding this issue.

Vm Configuration.

 

[fschauer]

This should be fixed with proxmox kernel version 6.2.16-12.

apt install proxmox-kernel-6.2.16-12-pve

Also check this thread for more: https://forum.proxmox.com/threads/vms-freeze-with-100-cpu.127459

 

[insightful]

After installing this reboot is required?
Because many VMS are running on this pve.
And one more question it will help for windows VMs?.
Thanks

 

[fschauer]

Yes, this newer kernel should also fix the 100% CPU freeze bug for Windows VMs.

A reboot will cause the new kernel to be booted.
In case you cannot reboot your host, you could take a look at KernelCare for kernel live patching. Unfortunately they only have support up to Proxmox VE 7 for now.

 

[insightful]

Hi,
I am getting this error when I am trying to Install-

Yes, this newer kernel should also fix the 100% CPU freeze bug for Windows VMs.

A reboot will cause the new kernel to be booted.
In case you cannot reboot your host, you could take a look at KernelCare for kernel live patching. Unfortunately they only have support up to Proxmox VE 7 for now.

 

[fschauer]

Did you run apt update beforehand?

 

[insightful]

Yes I have installed apt install proxmox-kernel-6.2.16-12-pve.
Again Faced vm freeze issue but this time CPU was normal.
In event viewer got this warning when windows was got down nothing was working Error:- Reset to device, \Device\RaidPort0, was issued.
Could you please help me to resolve this ?
root@pve2:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2: 6.2.16-12
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.7
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-5
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1
===========================================================================
agent: 1
balloon: 20480
boot: order=ide2;ide0;scsi0;net0
cores: 12
cpu: host
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
ide2: none,media=cdrom
machine: pc-i440fx-8.0
memory: 122880
meta: creation-qemu=8.0.2,ctime=1691771366
name: EFLEET-2019
net0: virtio=DE:92:76:29:4B:63,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
protection: 1
scsi0: ZFS-Data:vm-220-disk-0,iothread=1,size=100G
scsi1: ZFS-Data:vm-220-disk-1,iothread=1,size=2000G
scsihw: virtio-scsi-single
smbios1: uuid=84bf8ff0-0234-46aa-ad89-7511edb57168
sockets: 1
vmgenid: 81becdd1-80bf-41e0-b543-3ffd5c58910b

Thanks.

 

[Taomyn]

I'm having a similar issue with my Windows Server 2022 VMs, they randomly reboot during local vzdump backups, and I applied the newer kernel the other day. I have the same reset device event logged on all the servers:

root@stu:~# pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-12-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-12-pve: 6.2.16-12
proxmox-kernel-6.2: 6.2.16-12
proxmox-kernel-6.2.16-10-pve: 6.2.16-10
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.8
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-5
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1
root@stu:~# uname -r
6.2.16-12-pve

root@stu:~# qm config 101
agent: 1
balloon: 2048
bios: ovmf
boot: order=scsi0
cores: 2
cpu: host
efidisk0: cpool-01:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hotplug: disk,network,usb,memory,cpu
machine: pc-q35-8.0
memory: 4096
meta: creation-qemu=8.0.2,ctime=1692891555
name: SELMA
net0: virtio=0A:52:7B:37:D3:5B,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win11
scsi0: cpool-01:vm-101-disk-1,cache=writeback,discard=on,iothread=1,size=80G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=d9802007-9108-4e71-887c-143523931f00
sockets: 1
startup: order=1,up=120
vga: qxl
vmgenid: c6c99b82-9bc6-425b-b306-2561223767c2

My PVE is a very new installation, just a few weeks old so hopefully it hasn't gained too much baggage in that time that could be affecting it.

 

[Olli G.]

Any news on that issue?

 

[Taomyn]

Strangely I've not seen the issue since I reported my experiences and the only thing I have done was to limit the SMB share that the backups used with the option "cache=none" - I had noticed that sometimes backup files were still being "transferred" at the same time as others from earlier on, so it was overwhelming things at both ends (PVE and Windows Server). I've also done all the latest updates including the new VirtIO drivers that were just release. Cross fingers I didn't just jinx it.

 

[fiona]

Hi,

Any news on that issue?

if you are experiencing a similar issue, please share the output of pveversion -v, the VM configuration and some information about your host, e.g. CPU, RAM, used storages, etc.. Anything interesting in the system logs/journal around the time the issue happens?