Virtual machine freezes when shutting down (or rebooting)

[Vasily Poterjyko]

Windows 10 virtual machine freezes when shutting down (or rebooting). When this happens, one VM core is ~100% loaded.
The display of the virtual machine is black. The RESET function on the virtual machine does not lead to any result. Only STOP and then START.

Also with the same machine, the problem is, if you enable Memory Hotswap, then I get BSOD MEMORY_MANAGEMENT when loading vm. If you disable memory hotswap, then this problem does not exist. Maybe it has something to do with it, I don't know.

Any ideas?

Guest additions installed v0.1.208 or 0.1.215

Spoiler: qemu-server.conf

acpi: 1
agent: 1,fstrim_cloned_disks=1
balloon: 2048
bios: ovmf
boot: order=scsi0;scsi1
cores: 4
hotplug: disk,network,usb
machine: pc-i440fx-6.1
memory: 4096
meta: creation-qemu=6.1.0,ctime=1637516654
name: vm-win10
net0: virtio=0A:FC:03:61:25:7C,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: rpool-ssd:vm-6107-disk-0,size=64G,ssd=1
scsi1: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=b99030a9-5085-4629-b9fb-5a2a1b7ef5a1
sockets: 2
vmgenid: aca9c8a7-88cb-40d3-885e-f79fb1cc6aed

Spoiler: pveversion

proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-1-pve: 5.13.19-3
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

UPD:
There are quite a few such virtual machines. But freezes do not always occur, but like that spontaneously. Here's another machine on a different hypervisor that hung like this on reboot.

Spoiler: qemu_server.conf

agent: 1,fstrim_cloned_disks=1
balloon: 2048
bios: ovmf
boot: order=virtio0;ide2
cores: 4
cpu: host
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
machine: pc-i440fx-6.1
memory: 4096
name: vm-rg
net0: virtio=7A:7E:6D:9C:87:C4,bridge=vmbr2,firewall=1
net1: virtio=EE:9E:92:CB:D2:4A,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=90623e6d-2448-42f5-969c-6e57fe906318
sockets: 2
startup: order=2010,up=2,down=15
tablet: 1
vcpus: 4
vga: std
virtio0: local-zfs:vm-2010-disk-0,cache=writeback,size=64G
vmgenid: 282e1b69-d42c-4c5b-b56f-ef7a640e30bc

 

[nyviktor]

I have same problem, but did not find solution yet...

 

[nyviktor]

Windows 10 virtual machine freezes when shutting down (or rebooting). When this happens, one VM core is ~100% loaded.
The display of the virtual machine is black. The RESET function on the virtual machine does not lead to any result. Only STOP and then START.

Also with the same machine, the problem is, if you enable Memory Hotswap, then I get BSOD MEMORY_MANAGEMENT when loading vm. If you disable memory hotswap, then this problem does not exist. Maybe it has something to do with it, I don't know.

Any ideas?

Guest additions installed v0.1.208 or 0.1.215

Spoiler: qemu-server.conf

acpi: 1
agent: 1,fstrim_cloned_disks=1
balloon: 2048
bios: ovmf
boot: order=scsi0;scsi1
cores: 4
hotplug: disk,network,usb
machine: pc-i440fx-6.1
memory: 4096
meta: creation-qemu=6.1.0,ctime=1637516654
name: vm-win10
net0: virtio=0A:FC:03:61:25:7C,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: rpool-ssd:vm-6107-disk-0,size=64G,ssd=1
scsi1: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=b99030a9-5085-4629-b9fb-5a2a1b7ef5a1
sockets: 2
vmgenid: aca9c8a7-88cb-40d3-885e-f79fb1cc6aed

Spoiler: pveversion

proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-1-pve: 5.13.19-3
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

View attachment 33831


UPD:
There are quite a few such virtual machines. But freezes do not always occur, but like that spontaneously. Here's another machine on a different hypervisor that hung like this on reboot.
View attachment 33836

Spoiler: qemu_server.conf

agent: 1,fstrim_cloned_disks=1
balloon: 2048
bios: ovmf
boot: order=virtio0;ide2
cores: 4
cpu: host
hotplug: disk,network,usb,cpu
ide2: none,media=cdrom
machine: pc-i440fx-6.1
memory: 4096
name: vm-rg
net0: virtio=7A:7E:6D:9C:87:C4,bridge=vmbr2,firewall=1
net1: virtio=EE:9E:92:CB:D2:4A,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=90623e6d-2448-42f5-969c-6e57fe906318
sockets: 2
startup: order=2010,up=2,down=15
tablet: 1
vcpus: 4
vga: std
virtio0: local-zfs:vm-2010-disk-0,cache=writeback,size=64G
vmgenid: 282e1b69-d42c-4c5b-b56f-ef7a640e30bc

Change machine from i440fx to q35. It' work for me.

 

[Vasily Poterjyko]

Change machine from i440fx to q35. It' work for me.

It didn't help me. I set q35, loaded the virtual machine. Updated windows devices. I ran a reboot of the guest operating system and the system hung just like the previous time.

 

[nyviktor]

It didn't help me. I set q35, loaded the virtual machine. Updated windows devices. I ran a reboot of the guest operating system and the system hung just like the previous time.

Do you use the QEMU Guest Agent?

 

[Vasily Poterjyko]

Do you use the Use QEMU Guest Agent?

Yes, I have it enabled in the settings and installed in the guest system QEMU GUEST AGENT.

 

[peteb]

I also have the same problem after the most recent proxmox updates.
I cannot start windows 10 or windows 11 vm's on proxmox.

 

[Vasily Poterjyko]

With a dozen machines running windows10, after updating windows, you have to manually stop "press the STOP button" and then start it. These machines cannot reboot themselves, they freeze at the moment when the reboot itself should be. At the same time, one processor core is loaded at 100% according to the UI PVE indication.
This causes a lot of trouble.

Any ideas on my question?

 

[unluckywin2]

I have exactly the same problem on my side with 3 servers...

Any solution or ideas ?
This problem is only with Windows, I have many linux VM without problems.

One VM configuration example :

 

[Vasily Poterjyko]

I am still seeing this issue. I was under the impression that the problem is only on virtual machines with uefi bios.

 

[Keyrate]

I came here to see if others were experiencing this kind of problem w/ UEFI.

In my situation, it was not a Win10/Win11 machine, but FreeBSD (pfSense ... v2.5.2, 2.6.0 and pfSense+ 22.01). No matter what I would do (other than Start or Stop), whether it was reset, reboot from the Proxmox console, or ssh-ing to the machine and invoking a reboot from w/in the machine, the machine would not actually reboot and would hit 100% CPU utilization (I have only dedicated 1 CPU core, so that's easy to do). Often, if I realized that I've tried to shutdown or restart the VM rather than Stop, trying a stop afterwards will fail because the previous command will timeout leaving a /var/lock/qemu-server/lock-<vmid>.conf behind. Ultimately I'd have to either remove that file, or kill the kvm -id <vmid> associated w/ that VM to stop it.

I thought it might be that Proxmox couldn't manage it properly because I had configured QEMU Guest Agent initially, and pfSense (until 2.6.0) didn't support QEMU Guest Agent installation. Imagine my surprise and frustration after upgrading that I still had problems rebooting the system.

Like others here, I changed to the q35 machine time. That didn't resolve the problem. The only thing that did was switching from OVMF to SeaBIOS. Now everything works fine ... invoking GUI commands to shutdown, reboot or pause the VM; issuing qm commands on the command-line, or rebooting the machine within the VM itself.

I suspect there are a number of UEFI weirdnesses, as I ran across another weird UEFI specific problem that switching to SeaBIOS seems to have resolved. Obviously that's not a workaround for Win10/Win11 or other systems that benefit or expect UEFI. BTW, I tried 2 different builds of UEFI images other than the Proxmox-provided firmware, and I still had the problems. I have not tried a vanilla Linux / Virtmanager or other type-1 Hypervisor system (like TrueNAS Scale) to see if this is just a KVM/UEFI thing or something particular to Proxmox.

 

[Vasily Poterjyko]

After a short rest, I again returned to the analysis of this problem.
I cloned a virtual machine that was configured with UEFI. The configuration of this machine did not have an EFI drive connected. I created such an EFI disk in the GUI and started the virtual machine. The noVNC screen said "Guest has not initialized the display (yet)". At the same time, one core of the virtual machine was loaded at 100%. Thus, I got the situation about which I wrote at the very beginning. Only now I get the described hang every time the guest boots.

After the received result I have made one more test.
I created a new virtual machine, specifying Windows10, Bios-OVMF, Machine q35-v6.1 as OS, while adding an EFI disk before turning it on for the first time. The operating system booted successfully, without any of the above problems.

Thus, I assume that the cause of the described problem is that the guest was booted from the OVMF BIOS without an EFI drive connected. After that, there were some irreparable changes that led to the above problem.

Any thoughts on fixing the situation on such problematic guests without an EFI disk with an OVMF bios?

 

[LudoSmellsBad]

I have two windows 2019 VM, s that were having the same issues. After reading @Vasily Poterjyko post, I went back and add an EFI disk to both, and now both reboot fine. To be clear this is rebooting from within the VM's manually and rebooting from Windows Admin center.

 

[Vasily Poterjyko]

In virtual machines that use OVMF (UEFI) Bios, it is necessary that an EFI disk be added. If such a disk has not been added, then the problems described in the branch are observed. To solve this problem, you need to add an EFI disk. When adding a disk, you must uncheck the "Pre-Enroll Keys" option.

 

[Yosef_G]

In virtual machines that use OVMF (UEFI) Bios, it is necessary that an EFI disk be added. If such a disk has not been added, then the problems described in the branch are observed. To solve this problem, you need to add an EFI disk. When adding a disk, you must uncheck the "Pre-Enroll Keys" option.

Worked great! I think the problem with key of disks. If you restore or add new disk - you need remove EFI and add new without "Pre-Enroll Keys" !
Working - reboot and power off successfully. Updates working great on WIN11

Regards to Vasily

 

[elurk]

Worked great! I think the problem with key of disks. If you restore or add new disk - you need remove EFI and add new without "Pre-Enroll Keys" !
Working - reboot and power off successfully. Updates working great on WIN11

Regards to Vasily

Hi! I think I have this same problem. Could you please tell me how to do this step by step to a VM with Windows 2022 Server which already has been running services for a couple of weeks and which I cant remove/replace already? Thank you.