VM shutdown, KVM: entry failed, hardware error 0x80000021

After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code:
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code:
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
 
Last edited:
After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code:
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code:
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
Do you have Intel-Microcode AND/OR Mitigations=off set?
This may help as it looks so far....
 
  • Like
Reactions: t.lamprecht
Do you have Intel-Microcode AND/OR Mitigations=off set?
This may help as it looks so far....
Hi,

thank you for feedback.

No, i haven´t.

Where and how can i set Intel-Microcode AND/OR Mitigations=off please?

Thanks.
 
Hi,

thank you for feedback.

No, i haven´t.

Where and how can i set Intel-Microcode AND/OR Mitigations=off please?

Thanks.
On every Node edit /etc/default/grub and edit
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"
Save and quit Editor "nano" and enter "Update-Grub". The system should take care of the rest itself.
Reboot is needed.

For Intel-Microcode you need to add/change repository to include "non-free" and install it with "apt install intel-microcode". Reboot after.
 
  • Like
Reactions: rursache
Now I'm switched new kernel (5.15.35-1-pve) with "mitigations=off". I'm testing few days after inform you.
Wish me luck.
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
 
Last edited:
  • Like
Reactions: ITT
Is there any possibility for an official fix without disabling mitigations or downgrading to an old kernel?
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
 
  • Like
Reactions: rursache
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
 
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks

Just echoing the above as well. Three hosts updated last week and immediately found that VM's were shutting down with the errors indicated in the logs.

One of them kept blue screening too.

Rolled back the kernel and all three hosts became stable again.

Cheers
 
Hi there is update my test:

After mitigations=off, My windows VM crash during live backup. now best way is old kernel for me
İt's like this thread : https://forum.proxmox.com/threads/v...ntry-failed-hardware-error-0x80000021.109093/


Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
 
  • Like
Reactions: rursache
Installing the Intel microcode and setting the mitigation parameter on the 5.15.35 core only delayed the error, now it occurs once every 2 days.
Testing on 5.13.
Hwinfo: Intel i5 11400
W2019
 
  • Like
Reactions: rursache
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
Sorry, I ask someone from the "Staff Member" to have an official voice from them on how and when an official fix will be released for this, I would say, big problem with VM windows 2022, which makes them practically unusable on Proxmox VE with the new kernel still available in Proxmox "enterprise" updates ...
So I'm wondering when this bug will be fixed and when this update will be released so that we don't have to use the old kernel which at the moment seems like the only stable thing to get these VMs working. Thank you
 
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me

@proxmox-Team:
This fix just decrease shutdown interval but problem persists on my site.
As W2K22 is quite new not so many proxmox admins using it which can explain reports are not so high.
This will change for sure as more admins migrating...
Windows Server before 2022 do not have this problem.

I plan to revert to previous kernel as mentioned this should be stable.
 
Last edited:
  • Like
Reactions: rursache
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
Sorry, a few days have passed now ... we are waiting for an official fix, can you tell us something please? it seems to me really serious that for a problem like this you have to wait so long for a fixl, what problems are there? why don't you tell us anything? thank you
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!