Windows VMs stuck on boot after Proxmox Upgrade to 7.0

[check-ict]

We run VM's on multiple servers with different hardware. Customers with their own servers have the same issue.

In our own cluster, we use Intel Xeon and AMD Epyc CPU's. Both have the same issues.

 

[dea]

Many thanks for the confirmation @check-ict. And now how can we proceed @mira & & Proxmox support team? The snapshot test you asked for (also me in a ticket that I opened directly) I would say that it has been passed. Do you need further tests? Personally I am willing to change the repositories from enterprise to community or test, but no, please, a new agony with more than 50 Windows vm in stuck.

 

[yottabyteman]

Nice work, your every analysis coincides with mine.
I am left with one question, uncertain. Is there a link with the hardware? Are there systems that are affected by this problem and others that are not? I use Lenovo servers (quite a few) and they all have this problem. Other Lenovo systems that I use (other than) with version 6.4 DO NOT have this problem. But ... is the cause strictly QEMU (as I think) or is it a hardware (and consequently firmware) + QEMU combination?

We use custom-built servers that use dual Intel Xeon CPUs and Intel Motherboards and we still have the issue. All versions from old to latest CPU and motherboard.

 

[OsvaldoP]

Hy

we have same problem on multiple Nodes.
Most of them are Supermicro with AMD EPYC 7352 CPU

kr

 

[yottabyteman]

@Moayad , @mira, @tom, @aaron, @oguz
This is all the staff that has replied to this thread.

Can we get an update on what is being done to fix this? You ask for the image and I assume you got it, but with no acknowledgment that you got the image. I would also like some feedback that you see the problem and are working on a solution that you think will be out in x time or you see the issue and are trying to figure out why they are freezing when Windows asks for a reboot. I don't mean to be pushy and we love the software, but we have plans to expand and I can't be tracking down 100+ machines that do not reboot properly each week because of windows updates.

 

[mira]

We do have access to a VM exhibiting this issue, and we're currently trying to find the cause.
So all I can currently say is that we're working on it.

Once we know more, we'll provide more information both in the bug tracker and here in this, and possibly other, threads.

 

[yottabyteman]

We do have access to a VM exhibiting this issue, and we're currently trying to find the cause.
So all I can currently say is that we're working on it.

Once we know more, we'll provide more information both in the bug tracker and here in this, and possibly other, threads.

Thanks that makes me feel better that you can see the problem and are actively working on it. THANK YOU for the update.

 

[dea]

We do have access to a VM exhibiting this issue, and we're currently trying to find the cause.
So all I can currently say is that we're working on it.

Once we know more, we'll provide more information both in the bug tracker and here in this, and possibly other, threads.

@mira the fact that you are working on the problem is the best news of the last six months. VERY VERY VERY VERY THANKS (I've run out of corners where my head bangs with every windows upgrade).

 

[weehooey-bh]

Nice work, your every analysis coincides with mine.
I am left with one question, uncertain. Is there a link with the hardware? Are there systems that are affected by this problem and others that are not? I use Lenovo servers (quite a few) and they all have this problem. Other Lenovo systems that I use (other than) with version 6.4 DO NOT have this problem. But ... is the cause strictly QEMU (as I think) or is it a hardware (and consequently firmware) + QEMU combination?

@dea we have experienced this on Lenovo and HP servers.

 

[showiproute]

Same issue here BUT what is interesseting:
I run three servers in total - two of them are using "real" server hardware (Xeon/Epyc CPUs & 128 GB ECC RAM). Those two are having issues with Windows VMs.
The 3rd server is on a remote location and is kind of a better gaming PC (32 GB regular RAM, i7-10700K). This one is not having any issues with Windows VMs.

 

[dea]

@mira ... I would like to raise a question absolutely without a note of controversy, just to examine the reality of the facts.
I find it incorrect on the part of the Proxmox team to stop support for Proxmox VE 6.4 on July 31st when the 8 month version 7.x has problems, and I don't tell one individual, many individuals who pay for the Enterprise license.

I do not find it correct, it would be correct to extend support to release 6.4 at least until 7.x is really at the same level as 6.4, which (in my opinion) is NOT currently.

It is my thought, which does not want to accuse anyone, I know perfectly well that you are doing a GREAT job !!!

 

[alexander@cloud]

@mira ... I would like to raise a question absolutely without a note of controversy, just to examine the reality of the facts.
I find it incorrect on the part of the Proxmox team to stop support for Proxmox VE 6.4 on July 31st when the 8 month version 7.x has problems, and I don't tell one individual, many individuals who pay for the Enterprise license.

I do not find it correct, it would be correct to extend support to release 6.4 at least until 7.x is really at the same level as 6.4, which (in my opinion) is NOT currently.

It is my thought, which does not want to accuse anyone, I know perfectly well that you are doing a GREAT job !!!

@martin

 

[mira]

Can anyone reproduce this with cpu type `kvm64`?
All I could find were reports with either `cpu: Westmere` or `cpu: host` or similar CPU-specific types.

 

[dea]

@mira : seems indifferent. Answer No. 34

 

[guzi]

Can anyone reproduce this with cpu type `kvm64`?
All I could find were reports with either `cpu: Westmere` or `cpu: host` or similar CPU-specific types.

I can confirm, same issue with kvm64

 

[mira]

Thank you both. It seems I missed that one when going through the posts.

 

[tuxis]

@mira ... I would like to raise a question absolutely without a note of controversy, just to examine the reality of the facts.
I find it incorrect on the part of the Proxmox team to stop support for Proxmox VE 6.4 on July 31st when the 8 month version 7.x has problems, and I don't tell one individual, many individuals who pay for the Enterprise license.

I do not find it correct, it would be correct to extend support to release 6.4 at least until 7.x is really at the same level as 6.4, which (in my opinion) is NOT currently.

It is my thought, which does not want to accuse anyone, I know perfectly well that you are doing a GREAT job !!!

I tend to agree with this. We have not upgraded customer cluster only because of this bug. Also not complaining that the bug exists, but I now get customers asking why we haven't upgraded yet since they see a notification in the GUI.

 

[itNGO]

It is not only Windows... we also have this on 2 Ubuntu 20.4 VMs which auto-reboot every Thursday after installing security updates.... The always get stuck on "Rebooting" or "Black Screen"....

 

[mira]

I tend to agree with this. We have not upgraded customer cluster only because of this bug. Also not complaining that the bug exists, but I now get customers asking why we haven't upgraded yet since they see a notification in the GUI.

Please note that we will always try our best when it comes to support, even of unsupported versions.

Bugs won't be fixed, unless it is a simple backport from the current supported version.
And kernel updates will most likely not be available. So any security vulnerability affecting kernel 5.4 will most likely not be patched.

But other than that, we still try our best to help with any issues on unsupported versions.

 

[dea]

Please note that we will always try our best when it comes to support, even of unsupported versions.

Bugs won't be fixed, unless it is a simple backport from the current supported version.
And kernel updates will most likely not be available. So any security vulnerability affecting kernel 5.4 will most likely not be patched.

But other than that, we still try our best to help with any issues on unsupported versions.

Thanks @mira, I understand that supporting multiple versions becomes a hardly sustainable job. I think this specific case is a bit different, I have been using Proxmox since version 2.0 and I have never had a blocking problem that is not solved in 8 months. I also understand for us users who pay you a license, it becomes difficult to justify to our customers that their VMs go hung up to reboot once a month, try to understand why many of us still want to use Proxmox 6.4 because it presents no problems.

Apart from everything, I saw that in the devel branch you are working on Qemu 7.0. Have you identified something that can lead you to understand the cause of the problem?