Why EFI Windows VM sometimes gets stuck at boot?

[Darkbotic]

Hello everyone,

I've been noticing that sometimes, my Windows VM's with an EFI boot on Proxmox sometimes fail to fully boot/load.
When that happens, they always get stuck on this exact same screen:




This happens on different Proxmox nodes (not clustered) with different versions of Windows (10, 11 and Server).
Why does that happen?

Thanks in advance!

 

[frankmanzhu]

I have the same issue. I thought you had the fix as the thread is marked as "Tutorial"

 

[Darkbotic]

I have the same issue. I thought you had the fix as the thread is marked as "Tutorial"

Oh! Thanks. I messed up that. Fixed.
Hopefully someone from the staff can help us.

 

[itNGO]

Post VM Config please....

 

[Darkbotic]

First node:

proxmox-ve: 8.2.0 (running kernel: 6.8.12-2-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-2
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240813.2
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.3
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.1
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.13-2
pve-ha-manager: 4.0.5
pve-i18n: 3.2.3
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

agent: 1
bios: ovmf
boot: order=sata0
cores: 4
cpu: host
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-7.2
memory: 8192
meta: creation-qemu=7.2.0,ctime=1687539224
name: Windows
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
sata0: local-lvm:vm-102-disk-1,cache=writeback,discard=on,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=922dcc76-3fa8-3d36-bc18-f5a2df18ac3
sockets: 1
tpmstate0: local-lvm:vm-102-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: 5aad531a-6a3f-41cd-bcf8-637f5ca4c17a

 

[Darkbotic]

Second node:

proxmox-ve: 8.2.0 (running kernel: 6.8.12-2-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-2
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240813.2
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.3
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.1
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.13-2
pve-ha-manager: 4.0.5
pve-i18n: 3.2.3
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

agent: 1
bios: ovmf
boot: order=scsi0;net0;ide3
cores: 6
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-8.0
memory: 16384
meta: creation-qemu=8.0.2,ctime=1694531054
name: Windows
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=eba9da2c-a526-49b1-8f54-c6ab7aacee2f
sockets: 1
tpmstate0: local-lvm:vm-100-disk-2,size=4M,version=v2.0
vmgenid: ec4ba2ec-ffb1-421d-a49b-59734a5b7396

agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 6
cpu: host
efidisk0: local-lvm:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-8.1
memory: 8192
meta: creation-qemu=8.1.5,ctime=1709185049
name: Win10
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
scsi0: local-lvm:vm-102-disk-1,cache=writeback,discard=on,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=2b300271-5891-1a4d-a783-85b1136d9cab
sockets: 1
tpmstate0: local-lvm:vm-102-disk-2,size=4M,version=v2.0
vmgenid: 381dbc5a-8e65-48a6-b8ba-41375a979ebf

 

[Darkbotic]

Third node:

proxmox-ve: 8.2.0 (running kernel: 6.8.12-2-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-2
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
dnsmasq: 2.89-1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
intel-microcode: 3.20240813.2
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.3
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.1
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.13-2
pve-ha-manager: 4.0.5
pve-i18n: 3.2.3
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 12
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-9.0
memory: 16384
meta: creation-qemu=9.0.0,ctime=1719856358
name: WinServer
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,iothread=1,size=100G
scsi1: Storage:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=819221M
scsihw: virtio-scsi-single
smbios1: uuid=5bc37b35-7191-43cf-82ab-a496b998274c
sockets: 1
tpmstate0: local-lvm:vm-100-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: f79f7c13-42fd-41d5-935b-a0a3914bc4d9

agent: 1
bios: ovmf
boot: order=scsi0;ide0;net0
cores: 12
cpu: host
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
machine: pc-q35-9.0
memory: 8192
meta: creation-qemu=9.0.0,ctime=1719861005
name: Win11
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsi0: local-lvm:vm-101-disk-1,cache=writeback,discard=on,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=4167d2fb-8afc-4a04-bcd0-1870befc117c
sockets: 1
tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: a74a0b9a-51bc-4934-acdf-ba567f55d226

 

[itNGO]

And all VMs have this problem?

 

[Darkbotic]

I have more VM's on each of those nodes but I only included the QM Config output for the VM's that are having the issue. All other VM's in the nodes are based on Linux and they're not EFI. They're using the Default SeaBIOS. Also, just to clarify this behavior doesn't happen all the time maybe 1 to 3 times per month.

 

[Darkbotic]

Can anyone from staff help?

 

[_gabriel]

Try disabling the writecache , each disk, each VM.

 

[Darkbotic]

Try disabling the writecache , each disk, each VM.

Thanks.
I tried this but it didn't work.

 

[itNGO]

Do they never boot? Have you waited an hour or so if it ever continues the boot?

 

[Darkbotic]

Do they never boot? Have you waited an hour or so if it ever continues the boot?

It never boots when that happens, it stays stuck.
To fix it I have to stop the VM and then Start it again.

Is there any way to "monitor" from Proxmox when a VM doesn't boot. For example, when it doesn't respond to a ping?

 

[gfngfn256]

I notice you use the machine type pc-q35 on all your Window VMs. Do you have a specific reason for this?
Why not try the Default (i440fx) type.

 

[Darkbotic]

I notice you use the machine type pc-q35 on all your Window VMs. Do you have a specific reason for this?
Why not try the Default (i440fx) type.

Because pc-q35 has better PCI-E and GPU support
Do you think q35 is causing this issue?

 

[gfngfn256]

Because pc-q35 has better PCI-E and GPU support

Didn't notice any PCI/GPU passthrough on any of your VM's configs.

Do you think q35 is causing this issue?

I can't know for sure. But certainly worth a try using the (now) Default (i440fx) type.

IDK your CPU HW, but you may also want to try changing the CPU to the x86-64-v2-AES type, to see if it helps.


Please note, that if you have authenticated a Windows VM, some of the above changes may cause a need to reauthenticate.

 

[Darkbotic]

Didn't notice any PCI/GPU passthrough on any of your VM's configs.


I can't know for sure. But certainly worth a try using the (now) Default (i440fx) type.

IDK your CPU HW, but you may also want to try changing the CPU to the x86-64-v2-AES type, to see if it helps.


Please note, that if you have authenticated a Windows VM, some of the above changes may cause a need to reauthenticate.

I changed it to i440fx and x86-64-v2-AES and it broke Windows as you warned so I changed it back to q35 (I didn't need to change ), made a backup of the important files, deleted the VM and then installed Windows with i440fx and x86-64-v2-AES. Sadly, it happened again.

It looks like q35 and x86-64-v3 is not what's causing this.

 

[Petbotson]

Today i ran into the same issue with a Windows 2019 VM.

Sadly for me the issue is no fixed after hard stopping the vm and starting it again.

agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 8
cpu: x86-64-v3
description: ## Server
efidisk0: CC-ST03-OS-PVE:vm-133-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: none,media=cdrom
machine: pc-q35-9.0
memory: 16384
meta: creation-qemu=9.0.0,ctime=1723809789
name: virtual-server-name
net0: virtio=BC:24:11:16:CD:39,bridge=PRACLOUD,firewall=1
numa: 0
ostype: win10
scsi0: CC-ST03-OS-PVE:vm-133-disk-1,discard=on,iothread=1,size=200G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=77df56a9-94c0-46a6-8036-c11ee563d036
sockets: 1
tags:
vmgenid: 6b11847f-09bf-4306-afaf-d658ca9faaab

Did anyone ever figure this out? Or maybe can point me how to boot the VM again?

 

[Darkbotic]

I haven't figured it out. It only happens to the VM's using EFI regardless of the operating system.

I had to run a Kuma Docker Container to get a notification everytime any of them don't respond to a ping and then I have to log in to reset them manually.

I'm surprised that the staff haven't seen this issue as critical. @Petbotson since you're a subscriber, can you do us a favor and contact the official support and maybe help us with the solution they provide?