VM shutdown, KVM: entry failed, hardware error 0x80000021

Younex

Active Member
Jan 30, 2018
85
17
28
42
After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code:
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code:
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
 
Last edited:

itNGO

Well-Known Member
Jun 12, 2020
573
126
48
44
Germany
it-ngo.com
After upgrading my 5 node cluster to latest version i experience the same proble across all my Win2022 Server VM in randomly order shutting down at night with error:

Code:
 "KVM: entry failed, hardware error 0x80000021"

Obviously an kernel issue.

I revert to previous kernel.

Code:
proxmox-boot-tool kernel list
proxmox-boot-tool kernel pin 5.13.19-6-pve
reboot
Do you have Intel-Microcode AND/OR Mitigations=off set?
This may help as it looks so far....
 
  • Like
Reactions: t.lamprecht

itNGO

Well-Known Member
Jun 12, 2020
573
126
48
44
Germany
it-ngo.com
Hi,

thank you for feedback.

No, i haven´t.

Where and how can i set Intel-Microcode AND/OR Mitigations=off please?

Thanks.
On every Node edit /etc/default/grub and edit
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"
Save and quit Editor "nano" and enter "Update-Grub". The system should take care of the rest itself.
Reboot is needed.

For Intel-Microcode you need to add/change repository to include "non-free" and install it with "apt install intel-microcode". Reboot after.
 
  • Like
Reactions: rursache

kyesil

New Member
Nov 14, 2021
10
5
3
24
Now I'm switched new kernel (5.15.35-1-pve) with "mitigations=off". I'm testing few days after inform you.
Wish me luck.
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
 
Last edited:
  • Like
Reactions: ITT

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
5,511
1,760
164
South Tyrol/Italy
shop.proxmox.com
Is there any possibility for an official fix without disabling mitigations or downgrading to an old kernel?
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
 
  • Like
Reactions: rursache
Apr 27, 2016
12
10
23
53
www.altrove.info
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
 

Adam86

Member
Jul 11, 2018
12
0
21
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks

Just echoing the above as well. Three hosts updated last week and immediately found that VM's were shutting down with the errors indicated in the logs.

One of them kept blue screening too.

Rolled back the kernel and all three hosts became stable again.

Cheers
 

kyesil

New Member
Nov 14, 2021
10
5
3
24
Hi there is update my test:

After mitigations=off, My windows VM crash during live backup. now best way is old kernel for me
İt's like this thread : https://forum.proxmox.com/threads/v...ntry-failed-hardware-error-0x80000021.109093/


Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me
 
  • Like
Reactions: rursache

flamelcadet

New Member
Jun 8, 2022
1
1
1
Installing the Intel microcode and setting the mitigation parameter on the 5.15.35 core only delayed the error, now it occurs once every 2 days.
Testing on 5.13.
Hwinfo: Intel i5 11400
W2019
 
  • Like
Reactions: rursache
Apr 27, 2016
12
10
23
53
www.altrove.info
I also had the same problem on multiple proxmox VE servers of various customers with windows 2022 that turned off every night with the error indicated in the previous posts, at the moment I solved it using the old Kernel (5.13.19-6-pve) and everything started working again.
So I wonder when an official fix will come out for this blocking problem, without having to manually apply what is indicated here in the various posts and be able to use the latest Kernel without problems, thanks
Sorry, I ask someone from the "Staff Member" to have an official voice from them on how and when an official fix will be released for this, I would say, big problem with VM windows 2022, which makes them practically unusable on Proxmox VE with the new kernel still available in Proxmox "enterprise" updates ...
So I'm wondering when this bug will be fixed and when this update will be released so that we don't have to use the old kernel which at the moment seems like the only stable thing to get these VMs working. Thank you
 

Younex

Active Member
Jan 30, 2018
85
17
28
42
Ok guys. It's work for me 2+ days no crashes.

I want to share My fixing steps with you:

Note: My host : PVE 7.2 (new kernel 5.15.35-1-pve), my guest what crashes randomly: windows server 2022

1. step install Intel-Microcode install steps here: https://wiki.debian.org/Microcode

2. step: set mitigations=off
Code:
nano /etc/default/grub
find&change this line:
  GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"

update-grub2
reboot

solved my problem.

Please Note that mitigations=off coming with some security issues for the host . look here: https://unix.stackexchange.com/questions/554908/disable-spectre-and-meltdown-mitigations

Another way is downgrade PVE kernel to 5.13 but after some days it can be problem. because PVE's maybe stop old kernel support next releases.


maybe can help you.

Edit: if set mitigations=off, crash during live backup. now best way is old kernel for me

@proxmox-Team:
This fix just decrease shutdown interval but problem persists on my site.
As W2K22 is quite new not so many proxmox admins using it which can explain reports are not so high.
This will change for sure as more admins migrating...
Windows Server before 2022 do not have this problem.

I plan to revert to previous kernel as mentioned this should be stable.
 
Last edited:
  • Like
Reactions: rursache
Apr 27, 2016
12
10
23
53
www.altrove.info
Sure is, we actually waited on more positive feedback on the hunch raised, which was initially quite slow to pour in compared to the rate of complaints earlier, but now it looks quite good and allows us to confirm our theory.
Sorry, a few days have passed now ... we are waiting for an official fix, can you tell us something please? it seems to me really serious that for a problem like this you have to wait so long for a fixl, what problems are there? why don't you tell us anything? thank you
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!