Windows VMs stuck on boot after Proxmox Upgrade to 7.0
- Thread starter lolomat
- Start date
Hello everyone,
We had trouble with windows 10 VM.
This VM reboots every nigth at 12:00 and every reboot worked well. Yesterday night it rebooted for updating and in the morning I got the black windows page with circle..
very strange...
We had trouble with windows 10 VM.
This VM reboots every nigth at 12:00 and every reboot worked well. Yesterday night it rebooted for updating and in the morning I got the black windows page with circle..
very strange...
Has anyone had the exact same VM stuck twice?
I am asking because we purposely have left VMs running for extended periods of time (up to 65 days) without rebooting to attempt to learn about this issue. These are VMs that have experienced the problem in the past. There were on the exact same PVE nodes. When we rebooted them again, they did not hang.
We have rebooted them from the PVE GUI (presumably this is the QEMU agent interaction) and by applying updates with a scheduled reboot (Windows doing the reboot with no user logged in).
We have had 8-12 VMs with this issue but since we have started paying attention, the problem has not happened to the same VM twice.
I am asking because we purposely have left VMs running for extended periods of time (up to 65 days) without rebooting to attempt to learn about this issue. These are VMs that have experienced the problem in the past. There were on the exact same PVE nodes. When we rebooted them again, they did not hang.
We have rebooted them from the PVE GUI (presumably this is the QEMU agent interaction) and by applying updates with a scheduled reboot (Windows doing the reboot with no user logged in).
We have had 8-12 VMs with this issue but since we have started paying attention, the problem has not happened to the same VM twice.
Our experience has been the same as what others have reported -- after an automatic reboot from unattended windows updates, 2012R2 VMs stuck at the windows logo with the spinning dots. Have to stop and restart the VM, which someone pointed out in a ticket somewhere results in a "fresh" qemu process (IIRC).
Curious what happens if you live-migrate one of the windows VMs that's "spinning forever" -- that would move it to a "new" qemu process on the target node. I wonder if it would proceed to boot after the migration was complete without having to be shut down and started back up. I will try, if I can catch one happening in our environment.
Curious what happens if you live-migrate one of the windows VMs that's "spinning forever" -- that would move it to a "new" qemu process on the target node. I wonder if it would proceed to boot after the migration was complete without having to be shut down and started back up. I will try, if I can catch one happening in our environment.
@Jonathan Hankins That is an interesting idea to Live Migrate it. Definitely will try that.
Have you had any VMs hang more than once?
We see most of the time that guest has black screen after reboot or sometimes "guest has not initialized screen" and rarely forever spinning boot circle...
Ubuntu just stays on black screen here always....
Does not help.... just tried on a VM which hang after reboot....
it wouldn't even be useful as a workaround ... imagine a 9 node cluster with 300 Windows 2012 or higher VMs stuck ...
One thing I am wondering that has been little covered in this thread and that COULD be useful to understand better. Why does the problem NOT occur with 2008r2?
One thing I am wondering that has been little covered in this thread and that COULD be useful to understand better. Why does the problem NOT occur with 2008r2?
@aaron I saw that you were a proxmox staff member posting in this thread. I don't have a support contract, but if it's helpful, I've got a Win 2016 VM that just rebooted after windows updates and is experiencing the issue discussed in the thread. It's a dummy VM, so I can leave it this way if there is anything helpful I can generate for tech support to look at.
My Win2012r2 VMs and this Win2016 VM are experiencing this when auto-rebooting after Windows Updates. Not sure if I have ever seen it happen more than once on a given VM, as I only recently noticed it happening. I can confirm that live-migrating to a different node when it is in this state does NOT resolve the issue. If I run strace -f -p <pid_of_kvm_process> it doesn't appear to be hung, i.e., strace does spit out tons of output. The windows boot "spinning dots" progress indicator IS moving during the hang.
Pausing and resuming them VM does not fix it.
My Win2012r2 VMs and this Win2016 VM are experiencing this when auto-rebooting after Windows Updates. Not sure if I have ever seen it happen more than once on a given VM, as I only recently noticed it happening. I can confirm that live-migrating to a different node when it is in this state does NOT resolve the issue. If I run strace -f -p <pid_of_kvm_process> it doesn't appear to be hung, i.e., strace does spit out tons of output. The windows boot "spinning dots" progress indicator IS moving during the hang.
Pausing and resuming them VM does not fix it.
Same Issue, two different machines both with Windows Server 2019 VM's that works perfectly fine in PVE 6 now neither will reboot properly in PVE 7. I just get a black screen... the VM acts like it goes down ok, but it never comes back. I have several linux VM's and they seem to behave just fine. Just the windows ones. I have to go into the manager and stop the VM and then i can restart and get them to boot.
These VMs will not boot/reboot at all now? I do not believe anyone has experienced that. It may be a different issue.
This issue is usually a stuck VM, power it off, power up and it boots without issue. After that, VM will reboot normally.
One of the interesting things seems like it only happens once to a VM. We have not had confirmation of it happening more than once to any one VM.
thanks for the offer, but we are currently checking if we can fetch some VMs via our enterprise support anyway. Though I cannot say at this point if investigating such a VM further will actually offer some insights.
One thing I am curious about and would like to get some feedback is if anyone has the
pve-qemu-kvm package in version 6.2 installed. It is currently available in the no-subscription repository.Do some of you already use that version and if so, did you run into that issue again? Though it could need some time to trigger it again or be sure that it seems to have disappeared :-/
@aaron , i already reported, that with qemu 6.2 the issue is gone (on 2 of our Testservers).
Hope this helps.
I wish it were so, but it has been too little time since the release of qemu 6.2 in Proxmox. If the problem does not show up at the next Windows update in May, then yes it can be said to be solved. However ... Until qemu 6.2 is released in the enterprise branch many clusters will not use it. It is therefore necessary that the new version of qemu be certified for enterprise use, distribute it to all systems and verify the behavior. Only then can the problem be solved. It will take time.Thanks! Seems like I missed it
I'm using 6.2 now and the issue persists... and with new updates on ubuntu, I have the same issue now in linux. Maybe I wasnt clear in my last post... When i go to reboot the machines (now linux or windows) the machine goes down to a black screen... but never shuts off and never goes into a reboot. I have to force stop the machine and then start it, it will then boot as normal. So if I let windows reboot after updates... its hung and I have to force the machine down. This sounded like the same issues some of the others were having. But I also see there is an issue in here on booting the VM... I'm not currently having that issue they will boot fine, they just wont reboot or shutdown properly.
FWIW, the "stuck" VM I left running did NOT magically recover when I did rolling upgrades the other night and migrated it to a node with pve-qemu-kvm 6.2, it just kept spinning after the migration. I will keep track of reboots now that we are all on pve-qemu-kvm 6.2.