VM shutdown, KVM: entry failed, hardware error 0x80000021

After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code: 
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code: 
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
 
Last edited:
Younex said:
After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code: 
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code: 
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
Click to expand...
Do you have Intel-Microcode AND/OR Mitigations=off set?
This may help as it looks so far....
 
  • Like
Reactions: t.lamprecht
itNGO said:
Do you have Intel-Microcode AND/OR Mitigations=off set?
This may help as it looks so far....
Click to expand...
Hi,

thank you for feedback.

No, i haven´t.

Where and how can i set Intel-Microcode AND/OR Mitigations=off please?

Thanks.
 
Younex said:
Hi,

thank you for feedback.

No, i haven´t.

Where and how can i set Intel-Microcode AND/OR Mitigations=off please?

Thanks.
Click to expand...
On every Node edit /etc/default/grub and edit
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"
Save and quit Editor "nano" and enter "Update-Grub". The system should take care of the rest itself.
Reboot is needed.

For Intel-Microcode you need to add/change repository to include "non-free" and install it with "apt install intel-microcode". Reboot after.
 
  • Like
Reactions: rursache
kyesil said:
Now I'm switched new kernel (5.15.35-1-pve) with "mitigations=off". I'm testing few days after inform you.
Wish me luck.
Click to expand...
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code: 
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
 
Last edited:
  • Like
Reactions: ITT
alfe said:
Is there any possibility for an official fix without disabling mitigations or downgrading to an old kernel?
Click to expand...
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
 
  • Like
Reactions: rursache
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
 
Altrove said:
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
Click to expand...

Just echoing the above as well. Three hosts updated last week and immediately found that VM's were shutting down with the errors indicated in the logs.

One of them kept blue screening too.

Rolled back the kernel and all three hosts became stable again.

Cheers
 
Hi there is update my test:

After mitigations=off, My windows VM crash during live backup. now best way is old kernel for me
İt's like this thread : https://forum.proxmox.com/threads/v...ntry-failed-hardware-error-0x80000021.109093/


kyesil said:
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code: 
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
Click to expand...
 
  • Like
Reactions: rursache
Installing the Intel microcode and setting the mitigation parameter on the 5.15.35 core only delayed the error, now it occurs once every 2 days.
Testing on 5.13.
Hwinfo: Intel i5 11400
W2019
 
  • Like
Reactions: rursache
Altrove said:
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
Click to expand...
Sorry, I ask someone from the "Staff Member" to have an official voice from them on how and when an official fix will be released for this, I would say, big problem with VM windows 2022, which makes them practically unusable on Proxmox VE with the new kernel still available in Proxmox "enterprise" updates ...
So I'm wondering when this bug will be fixed and when this update will be released so that we don't have to use the old kernel which at the moment seems like the only stable thing to get these VMs working. Thank you
 
  • Like
Reactions: rursache, samuele, User01 and 3 others
kyesil said:
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code: 
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
Click to expand...

@proxmox-Team:
This fix just decrease shutdown interval but problem persists on my site.
As W2K22 is quite new not so many proxmox admins using it which can explain reports are not so high.
This will change for sure as more admins migrating...
Windows Server before 2022 do not have this problem.

I plan to revert to previous kernel as mentioned this should be stable.
 
Last edited:
  • Like
Reactions: rursache
t.lamprecht said:
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
Click to expand...
Sorry, a few days have passed now ... we are waiting for an official fix, can you tell us something please? it seems to me really serious that for a problem like this you have to wait so long for a fixl, what problems are there? why don't you tell us anything? thank you
 
  • Like
Reactions: MasinAD, rursache and IsThisThingOn
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back