hardware error 0x80000021

[sporeburst]

Hello.

First-time user of Proxmox. I've been experiencing the hardware error 0x80000021 issue, and I read the thread on how to fix it by disabling nested Virt and disabling mmu.

I have 11 VMs running Debian 11

proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)

------------------------------

Hardware(Yes, I know it's old):

SuperMicro X9DRi-LN4F+
Dual E5-2620 v2 (IvyBridge socket 2011)
768GB RAM

------------------------------

Fixes attempted:

1. Disabled mmu:

cat /sys/module/kvm/parameters/tdp_mmu N

2. Updated intel microcode on the Proxmox host. Do I need to do this in all of the VMs?

3. Changed processor type for each VM to "IvyBridge"

4. Added pcid, spec-ctrl, ssbd, aes on the processor options

5. Checked temp sensors: Cpu1: 38C Cpu2: 40C HD's: 35C

6. Running latest Supermicro BIOS Version

7. Disabled RAM Ballooning

-----------------------------

Other Info:

VM NICs are VirtIO(Paravirtualized)
VM Bios is Default(SEABios)
VM Display: Standard VGA
VM Machine: Default(i440fx)
VM SCSI Controller: VirtIO SCSI Single
Supermicro BIOS has Hyperthreading Enabled
Supermicro BIOS has VT-d enabled

------------------------------------


I'd like to attempt using Kernel 5.13 like the thread mentioned but I don't know how to do that, but I haven't read the docs just yet. Would this be something worth trying?

Thanks in advance.

 

[fiona]

Hi,
can you share the output of pveversion -v, the config of an affected VM qm config <ID> and the full error message from /var/log/syslog?

You can use apt install pve-kernel-5.13, but I'd recommend to try out a newer kernel first instead:
https://forum.proxmox.com/threads/opt-in-linux-6-1-kernel-for-proxmox-ve-7-x-available.119483/

The TDP issue has been solved in current kernels, so very likely your issue is different. From the Roadmap:

This has been adressed with the kernel package pve-kernel-5.15.39-3-pve with version 5.15.39-3 and newer, which is available on all repositories.

 

[sporeburst]

Hello fiona and thank you for your reply!

I performed some searched and found out how to downgrade kernels. I first downgraded to 5.13 and the issue still remained so I downgraded again to 5.11 where it still remained.

root@dev:~# pveversion -v proxmox-ve: 7.3-1 (running kernel: 5.11.7-1-pve) pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4) pve-kernel-helper: 7.3-2 pve-kernel-5.15: 7.3-1 pve-kernel-5.15.83-1-pve: 5.15.83-1 pve-kernel-5.15.74-1-pve: 5.15.74-1 pve-kernel-5.13.14-1-pve: 5.13.14-1 pve-kernel-5.11.7-1-pve: 5.11.7-1 ceph-fuse: 17.2.5-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.3 libproxmox-backup-qemu0: 1.3.1-1 libpve-access-control: 7.3-1 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.3-1 libpve-guest-common-perl: 4.2-3 libpve-http-server-perl: 4.1-5 libpve-storage-perl: 7.3-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.0-3 lxcfs: 4.0.12-pve1 novnc-pve: 1.3.0-3 proxmox-backup-client: 2.3.2-1 proxmox-backup-file-restore: 2.3.2-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.5.3 pve-cluster: 7.3-2 pve-container: 4.4-2 pve-docs: 7.3-1 pve-edk2-firmware: 3.20220526-1 pve-firewall: 4.2-7 pve-firmware: 3.6-2 pve-ha-manager: 3.5.1 pve-i18n: 2.8-1 pve-qemu-kvm: 7.1.0-4 pve-xtermjs: 4.16.0-1 qemu-server: 7.3-2 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+2 vncterm: 1.7-1 zfsutils-linux: 2.1.7-pve3

Being desperate, I disabled Hyperthreading in my Supermicro bios and all of my VMs have been stable for 12 hours. Now of course I've lost half of my threads though. Let me try upgrading to kernel 6.1, enable hyperthreading and see what happens. Typically the issue will present itself within the first 15 minutes of the VMs turning on.


Thanks

 

[sporeburst]

Didn't work. VM 5 had an internal error already:

root@dev:~# qm config 105 balloon: 0 boot: order=scsi0;net0 cores: 1 cpu: IvyBridge,flags=+aes memory: 16384 meta: creation-qemu=7.1.0,ctime=1674925143 name: xxxxxxxxx net0: virtio=12:3D:9C:F0:26:2E,bridge=vmbr1,firewall=1 numa: 1 ostype: l26 scsi0: StorageFour:vm-105-disk-0,iothread=1,size=3000G scsihw: virtio-scsi-single smbios1: uuid=71f170d6-a7c8-41cd-bc48-7e938938389d sockets: 2 vga: std vmgenid: a59560bc-22e3-4aa6-ac32-96a4b6b6cff0 root@dev:~#

root@dev:~# pveversion -v proxmox-ve: 7.3-1 (running kernel: 6.1.2-1-pve) pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4) pve-kernel-helper: 7.3-2 pve-kernel-5.15: 7.3-1 pve-kernel-6.1.2-1-pve: 6.1.2-1 pve-kernel-5.15.83-1-pve: 5.15.83-1 pve-kernel-5.15.74-1-pve: 5.15.74-1 pve-kernel-5.13.14-1-pve: 5.13.14-1 pve-kernel-5.11.7-1-pve: 5.11.7-1 ceph-fuse: 17.2.5-pve1 corosync: 3.1.7-pve1 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve2 libproxmox-acme-perl: 1.4.3 libproxmox-backup-qemu0: 1.3.1-1 libpve-access-control: 7.3-1 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.3-1 libpve-guest-common-perl: 4.2-3 libpve-http-server-perl: 4.1-5 libpve-storage-perl: 7.3-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 5.0.0-3 lxcfs: 4.0.12-pve1 novnc-pve: 1.3.0-3 proxmox-backup-client: 2.3.2-1 proxmox-backup-file-restore: 2.3.2-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.5.3 pve-cluster: 7.3-2 pve-container: 4.4-2 pve-docs: 7.3-1 pve-edk2-firmware: 3.20220526-1 pve-firewall: 4.2-7 pve-firmware: 3.6-2 pve-ha-manager: 3.5.1 pve-i18n: 2.8-1 pve-qemu-kvm: 7.1.0-4 pve-xtermjs: 4.16.0-1 qemu-server: 7.3-2 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.8.0~bpo11+2 vncterm: 1.7-1 zfsutils-linux: 2.1.7-pve3 root@dev:~#

Error message:

dev QEMU[1973]: KVM: entry failed, hardware error 0x80000021 dev QEMU[1973]: If you're running a guest on an Intel machine without unrestricted mode dev QEMU[1973]: support, the failure can be most likely due to the guest entering an invalid dev QEMU[1973]: state for Intel VT. For example, the guest maybe running in big real mode dev QEMU[1973]: which is not supported on less recent Intel processors. dev kernel: set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. dev QEMU[1973]: RAX=ffffffffa9f00f20 RBX=0000000000000000 RCX=ffff8b9e77c34940 RDX=0000000000007072 dev QEMU[1973]: RSI=0000000000000083 RDI=0000000000000083 RBP=ffffffffaac13940 RSP=ffffffffaac03eb8 dev QEMU[1973]: R8 =0000002d119de1c6 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000 dev QEMU[1973]: R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000 dev QEMU[1973]: RIP=ffffffffa9f0108e RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 dev QEMU[1973]: ES =0000 0000000000000000 ffffffff 00c00000 dev QEMU[1973]: CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] dev QEMU[1973]: SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] dev QEMU[1973]: DS =0000 0000000000000000 ffffffff 00c00000 dev QEMU[1973]: FS =0000 0000000000000000 ffffffff 00c00000 dev QEMU[1973]: GS =0000 ffff8b9e77c00000 ffffffff 00c00000 dev QEMU[1973]: LDT=0000 0000000000000000 ffffffff 00c00000 dev QEMU[1973]: TR =0040 fffffe0000003000 00004087 00008b00 DPL=0 TSS64-busy dev QEMU[1973]: GDT= fffffe0000001000 0000007f dev QEMU[1973]: IDT= fffffe0000000000 00000fff dev QEMU[1973]: CR0=80050033 CR2=0000562641f36010 CR3=000000005940a000 CR4=001506f0 dev QEMU[1973]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 dev QEMU[1973]: DR6=00000000ffff0ff0 DR7=0000000000000400 dev QEMU[1973]: EFER=0000000000000d01 dev QEMU[1973]: Code=cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d b6 8e 50 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00 00 0f 00 2d 96 8e 50 00

 

[fiona]

Being desperate, I disabled Hyperthreading in my Supermicro bios and all of my VMs have been stable for 12 hours. Now of course I've lost half of my threads though. Let me try upgrading to kernel 6.1, enable hyperthreading and see what happens. Typically the issue will present itself within the first 15 minutes of the VMs turning on.

Glad you were able to find a workaround

Didn't work. VM 5 had an internal error already:
root@dev:~# pveversion -v proxmox-ve: 7.3-1 (running kernel: 6.1.2-1-pve)

Since it didn't work on any kernels you tested, kvm with hyperthreading on that CPU might just not be fully supported. Or it might be a long-standing kernel/kvm bug.

What you can still try is different QEMU versions. For example

apt install pve-qemu-kvm=6.0.0-4

and

apt install pve-qemu-kvm=6.2.0-11