windows server 2022 VM black screen after reboot. No login screen but I can ping and access file shares

[genesispro]

Hi guys, I am having a serious issue that happened 2nd time and both time I haven't managed to recover unless I recover from a backup. I have a VM running windows server 2022 standard. I updated proxmox to 8.1.10 it was something like 8.1.4 before. I shut down the windows VM because there was a kernel update and I wanted to reboot the proxmox as well. Since the reboot the windows VM only boots some services like ping and file share (no RDP) and on the console all I can see is a black screen and I can't log in or ALT-CTRL-DEL to at least get a taskmanager up and running. Both VMs had been running for 5-6 months and both got stuck on a black screen after a proper windows reboot. Any ideas?

 

[genesispro]

something to note. it has happened to 2 totally different machines, in totally different datacenters and with a 2 months difference each other. The only relation I can think off is that I upgraded proxmox (but not too many versions I usually update it every 2-3 months). Both where at version 8, both where running window server 2022 standard and to both I did a proper shutdown in windows before I restarted the hypervisor

 

[genesispro]

I am trying to add as many information as possible while I am investigating. Next one is that zabbix agent was one of the services that was running reporting an Operating System title change.
from>>>2024-04-08 23:30:23 Windows Server 2022 Standard 20348.1.amd64fre.fe_release.210507-1500 Build 20348.2322
to>>>2024-04-09 04:30:23 Windows Server 2022 Standard 20348.1.amd64fre.fe_release.210507-1500 Build 20348.2340

 

[andaga]

Hello!
Can you share you conf file of the vm ?
Also when the vm is booted up you can see if the qemu-guest-agent is up into the proxmox summary tab?

 

[genesispro]

agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=virtio0;net0
cores: 16
cpu: host
machine: pc-q35-8.0
memory: 49152
meta: creation-qemu=8.0.2,ctime=1699899300
name: boatmenwin1
net0: virtio=D6:9E:9D:10:E7:3C,bridge=vmbr0,tag=70
numa: 0
onboot: 1
ostype: win11
protection: 1
scsihw: virtio-scsi-single
smbios1: uuid=bb07220c-99ee-4e66-bb4e-911b60c29edc
sockets: 1
tablet: 1
virtio0: nvme2tbpm03:vm-101-disk-0,iothread=1,size=200G
vmgenid: 77c6a3af-322d-467d-898c-4788a0137f65

#qmdump#map:virtio0:drive-virtio0:nvme2tbpm03:raw:

 

[andaga]

Which display you use for the console ( Try to use the HTML5 NoVNC).

 

[genesispro]

this is what I use -> Console->noVNC

 

[andaga]

During bootup of the VM it's displaying something and it's only the login screen black? Or it's everytime black screen?

 

[genesispro]

I see the circle the spins and right before the blue screen pops to display loading info or the login screen the spinner disappears and the black screen remains. In the meantime it pings and the shared files are accessible

 

[andaga]

The qemu-guest-agent is running? If yes you can try to debug from there

 

[genesispro]

how do I debug using the qemu guest agent? I didn't know I could

 

[gfngfn256]

I don't know how long either VM was running before a reboot, so it could be anything (Proxmox/Windows update) during that period.

Have you thought about starting the Windows VM in safe mode/Windows Recovery Environment (WinRE)? This maybe tricky to accomplish (maybe you have ssh access, or you could boot from media etc. (possibly even some keyboard trickery on the sign-in screen you can't see) - but should be possible. You may possibly then maybe able to boot with an actual workable console & discover which driver(s) need attention. Make sure you have proper/restorable backups for the VM.

 

[genesispro]

the production is running? I restored a backup from the PBS but since it is the second time that it happened I want to figure out why it is happening to avoid such issues in the future. I have tried several things from the recovery console... I tend to break it more that manage to fix it doing bcdedit etc. When I make it worse I restore the snapshot that I took to continue trying to spot it.

 

[gfngfn256]

Do you ever/often reboot the VM? Does it always reboot correctly?

Don't know what HW you are on, but maybe something you could try: set (in GUI) the VM (left-pane), Processors, Type to x86-64-v2-AES. This also has added benefit for migration - since it isn't local host HW specific.

Always make sure you have proper/restorable backups for the VM.

 

[genesispro]

1) I do. I have even rebooted to test the restored VM which was 1 day back. I reboot I shutdown fine!
2) I have tried using the cpu emulator you mentioned

 

[gfngfn256]

So your issue really needs addressing!

Is there any host GPU or other passthrough trickery going on, to another LXC or VM? (I know its unlikely on 2 VMs in different Datacenters - but still worth investigating).

What Display options have you tried? Default, Spice, VmWare comp. etc. ? What Machine options?

Edit: Sorry meant to ask: Whenever you reboot Proxmox Host, does VM always gracefully start?

 

[genesispro]

I have even backed it up and moved it to another proxmox hypervisor and still the same. In this new proxmox all VMs are stopped except this one

 

[genesispro]

it turns out the c:\windows\registration folder was missing. I recovered that folder and it booted but now a new question rises. How come when I shutted down windows normally and then rebooted proxmox hypervisor it ended up loosing that folder? Could it be that something was in the ZFS cache that was not committed to the disks and when I rebooted the hypervisor it was lost? If that is the case does proxmox ensures that all data is committed before it reboots?

 

[gfngfn256]

What is your cache setting (GUI) for your VM's disk?
What is your ZFS sync set to: zfs get sync {poolName} output?

 

[genesispro]

cache: Default (No cache)
io thread: true
backup: true
skip replication: false
Async IO: Default(io_uring)

root@dcpm03:~# zfs get sync nvme2tbpm03
NAME PROPERTY VALUE SOURCE
nvme2tbpm03 sync standard default