Search results

@andrewrf for all the VMs that have hung, are you seeing the following: Only Windows VMs, Server 2012 or later Windows does not report anything in bootlog (ntbtlog.txt) -- meaning it had not got far enough in booting to start that logging Only hangs on a reboot and only if the VM has been...

@Moayad Here is the same data from another hung VM. It is the "classic" black screen, Windows logo and endlessly spinning balls. Power off (Stop) the VM, power on and it started normally.

@tom No one that we are aware of has reported "Getting Windows Ready". That is a different problem. Has anyone reviewed the information we provided that was requested by @Moayad ?

I have posted a link in the bug report to the data we captured: https://bugzilla.proxmox.com/show_bug.cgi?id=3933

Hi @Moayad please see the attached files. This VM is currently hung. We can leave it for an hour or two before it needs to get it back into production. Please let us know ASAP if there is something we can do further. A few notes: This is the first time we have seen a screen like this...

@itNGO are all the VMs that repeatedly freeze using version 6.0 or later of i440fx? # qm showcmd <vmid> --pretty | grep machine -machine 'type=pc-i440fx-6.1+pve0' \ I noticed that in your video. Also, in PVE Roadmap I noticed under 6.4: Support pinning a VM to a specific QEMU machine...

Hey @tstrand have you had this issue recently? If so, was it the same VMs or different ones?

Hey @itNGO thanks for the video. Most of our VMs were built in PVE and have only experienced the hang once each (that we know of). I just rebooted one that had 49 days of uptime and it rebooted fine. It had been one of the ones that had a problem earlier. Earlier today, rebooted another one...

@itNGO how do you reproduce it? We have some VMs too but have not found a way to reproduce the hang.

@tkffaul you could replicate it?! Interesting. We have not been able to replicate it other than to upgrade a 6.x cluster to 7.x. After the upgrade, all (most?) Windows VMs will hang on the first reboot. After they have had the failed reboot once, it does not appear to happen again to the same...

@tkffaul have you had this happen more than once to each VM? We have not recorded the same VM having the problem more than once. However, we are not 100% confident that is the case.

These VMs will not boot/reboot at all now? I do not believe anyone has experienced that. It may be a different issue. This issue is usually a stuck VM, power it off, power up and it boots without issue. After that, VM will reboot normally. One of the interesting things seems like it only...

@Jonathan Hankins That is an interesting idea to Live Migrate it. Definitely will try that. Have you had any VMs hang more than once?

Has anyone had the exact same VM stuck twice? I am asking because we purposely have left VMs running for extended periods of time (up to 65 days) without rebooting to attempt to learn about this issue. These are VMs that have experienced the problem in the past. There were on the exact same PVE...

@vulture What do you need to do to get the test VM to hang? Asking because, after a hang, it seems to take a while before the same VM will hang again. Do you have to leave it for X days? Or is there something you can do to trigger it?

Hey @Emilien Sure we can try that. One of the VMs that froze has been changed to cpu: host. Some of these VMs have been running on the same hardware for over two years and only switched to cpu: Westmere about a year ago. This same cluster was only updated to 7.x this year when the hanging...

Hi Tuxis, Thank you for doing that!

@Whatever thanks for adding to the data. Some additional information we have found: - It is affecting both HA and non-HA VMs - When a VM is booting, at some point the PVE console window resizes (when the OS and hardware agree on resolution?). The hang happens before the console resizes.

What we have learned so far from our research and compiled from others in this forum post: - Windows does not report anything in bootlog (ntbtlog.txt) -- suggests issue happens very early in the boot process - Windows Server version does not appear to affect it -- 2016 to 2022 affected - Only...

We are seeing the same behaviour since upgrading to 7.x. At first, we thought it was Windows Updates because they always break and you usually only reboot a Windows Server if you are applying updates. A sample of the machines that have hung: Windows Server 2019 Datacenter (1809) agent: 1...