VMs won't start after upgrading pve-qemu-kvm

[djgraff209]

Hi,

I have encountered an issue after upgrading my Proxmox VE system.

VMs will not start if I'm using pve-qemu-kvm:amd64 greater than 6.0.0-3.

The latest updates included pve-qemu-kvm:amd64=6.1.0-1

I see the following in the syslog:

Nov 10 12:16:53 cloud pvedaemon[1194]: <root@pam> end task UPID:cloud:00000825:0                                                                         0002C3B:618BFE83:qmstart:106:root@pam: OK

Nov 10 12:16:56 cloud pvedaemon[1194]: VM 106 qmp command failed - VM 106 qmp command 'guest-ping' failed - got timeout
Nov 10 12:17:00 cloud systemd[1]: Starting Proxmox VE replication runner...
Nov 10 12:17:01 cloud CRON[2197]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Nov 10 12:17:01 cloud systemd[1]: pvesr.service: Succeeded.
Nov 10 12:17:01 cloud systemd[1]: Finished Proxmox VE replication runner.
Nov 10 12:17:01 cloud systemd[1]: pvesr.service: Consumed 1.190s CPU time.
Nov 10 12:17:02 cloud QEMU[2121]: KVM internal error. Suberror: 1
Nov 10 12:17:02 cloud QEMU[2121]: emulation failure
Nov 10 12:17:02 cloud QEMU[2121]: EAX=0000c1c1 EBX=000f000d ECX=00001234 EDX=0000db80
Nov 10 12:17:02 cloud QEMU[2121]: ESI=000efd40 EDI=000eb684 EBP=0000fe58 ESP=000eb658
Nov 10 12:17:02 cloud QEMU[2121]: EIP=0000ff68 EFL=00000087 [--S--PC] CPL=0 II=0 A20=1 SMM=1 HLT=0
Nov 10 12:17:02 cloud QEMU[2121]: ES =0000 00000000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: CS =a000 000a0000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: SS =0000 00000000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: DS =0000 00000000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: FS =0000 00000000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: GS =0000 00000000 ffffffff 00809300
Nov 10 12:17:02 cloud QEMU[2121]: LDT=0000 00000000 0000ffff 00008200
Nov 10 12:17:02 cloud QEMU[2121]: TR =0000 00000000 0000ffff 00008b00
Nov 10 12:17:02 cloud QEMU[2121]: GDT=     00008280 00000027
Nov 10 12:17:02 cloud QEMU[2121]: IDT=     00000000 00000000
Nov 10 12:17:02 cloud QEMU[2121]: CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
Nov 10 12:17:02 cloud QEMU[2121]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Nov 10 12:17:02 cloud QEMU[2121]: DR6=00000000ffff0ff0 DR7=0000000000000400
Nov 10 12:17:02 cloud QEMU[2121]: EFER=0000000000000000
Nov 10 12:17:02 cloud QEMU[2121]: Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.
Nov 10 12:17:02 cloud kernel: [  124.187676] vmbr0: port 3(tap106i0) entered disabled state
Nov 10 12:17:02 cloud kernel: [  124.188918] vmbr0: port 3(tap106i0) entered disabled state

I have seen this related to something called SMM being removed in (I believe QEMU) but I am not deeply skilled on the matter.

I'm running on a Dell R710 with two Xeon X5550 cpus.

 

[Moayad]

Hi,

Have you tried to change the CPU type on the VM to kvm64 or Host?

You can do that by going to PVE GUI -> Datacenter -> {NodeName} -> VM -> Hardware -> Processors (Edit it and change the CPU type to kvm64 or Host)

If that not helps please post the VM config qm config {VMID} and the output of pveversion -v

 

[t.lamprecht]

I have seen this related to something called SMM being removed in (I believe QEMU) but I am not deeply skilled on the matter.

It may be the other way around, we disabled SMM for a while as it made problems in the distant past but enabled it now (among other reasons for TPM/Win11 support).

Can you please post the VM config of the affected VM, e.g.: qm config 106

 

[djgraff209]

Output of qm config 106 (note I have omitted my SSH configuration)

agent: enabled=1
balloon: 4096
boot: c
bootdisk: scsi0
cores: 4
description: Debian Bullseye
ide2: local-images:106/vm-106-cloudinit.qcow2,media=cdrom,size=4M
ipconfig0: ip=dhcp
memory: 16384
name: spring-buildbox
net0: virtio=6A:22:97:F6:DA:36,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local-images:9003/base-9003-disk-0.qcow2/106/vm-106-disk-0.qcow2,size=16G
scsihw: virtio-scsi-pci
searchdomain: oneangrybean.lan
serial0: socket
smbios1: uuid=cc1b3023-64b2-4513-a189-303a09bd8970
sockets: 4
sshkeys: *** OMITTED ***
vga: serial0
vmgenid: 2d1d2e70-d74b-4ed2-a1e3-f5fa512512ff

Output of pveversion -e (note: I have manually downgraded pve-qemu-kvm to version 6.0.0-3 which is the only one currently working)

root@cloud:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-7-pve)
pve-manager: 7.0-14 (running version: 7.0-14/a9dbe7e3)
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-12
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-13
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.13-1
proxmox-backup-file-restore: 2.0.13-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-18
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

If there is additional data or information you require please let me know and thank you for your assistance.

 

[djgraff209]

Hi,

Have you tried to change the CPU type on the VM to kvm64 or Host?

You can do that by going to PVE GUI -> Datacenter -> {NodeName} -> VM -> Hardware -> Processors (Edit it and change the CPU type to kvm64 or Host)

If that not helps please post the VM config qm config {VMID} and the output of pveversion -v

I apologize - I forgot to mention - I have this currently as kvm64 - I have tried switching to Host with no positive result.

 

[djgraff209]

Monitoring the updates this morning I noticed a significant number of packages had changed.

I am happy to report that the latest set of updates seems to have remedied the problem.

Below is an excerpt from /var/log/apt/history.log:

Start-Date: 2021-11-18  06:39:53
Commandline: apt upgrade -y
Install: libjs-qrcodejs:amd64 (1.20201119-pve1, automatic), libposix-strptime-perl:amd64 (0.13-1+b7, automatic), pve-kernel-5.13.19-1-pve:amd64 (5.13.19-2, automatic), pve-kernel-5.13:amd64 (7.1-4, automatic)
Upgrade: pve-docs:amd64 (7.0-5, 7.1-2), proxmox-widget-toolkit:amd64 (3.3-6, 3.4-2), libpve-rs-perl:amd64 (0.2.3, 0.3.2), pve-qemu-kvm:amd64 (6.0.0-3, 6.1.0-2), libpve-cluster-api-perl:amd64 (7.0-3, 7.1-2), libpve-storage-perl:amd64 (7.0-13, 7.0-15), libpve-guest-common-perl:amd64 (4.0-2, 4.0-3), pve-cluster:amd64 (7.0-3, 7.1-2), proxmox-ve:amd64 (7.0-2, 7.1-1), proxmox-backup-file-restore:amd64 (2.0.13-1, 2.0.14-1), qemu-server:amd64 (7.0-18, 7.1-3), libpve-access-control:amd64 (7.0-6, 7.1-2), pve-container:amd64 (4.1-1, 4.1-2), pve-i18n:amd64 (2.5-1, 2.6-1), proxmox-backup-client:amd64 (2.0.13-1, 2.0.14-1), pve-manager:amd64 (7.0-14, 7.1-5), libpve-common-perl:amd64 (7.0-12, 7.0-14), pve-edk2-firmware:amd64 (3.20210831-1, 3.20210831-2), libpve-cluster-perl:amd64 (7.0-3, 7.1-2)
End-Date: 2021-11-18  06:41:13

Thank you for your efforts on this!

 

[t.lamprecht]

I am happy to report that the latest set of updates seems to have remedied the problem.

Great to hear that, thanks for reporting back!

I'd think the new 5.13 kernel was the remedy here, at least if you rebooted after the update.
Out of interest, which CPU model is this host using?

 

[Polyphemus]

I've got the same problem with one of my VM's, after upgrading to 7.1:

Nov 18 13:41:08 pve pvestatd[1139]: VM 117 qmp command failed - VM 117 qmp command 'query-proxmox-support' failed - unable to connect to VM 117 qmp socket - timeout after 31 retries

I tried host CPU and KVM64, but that does not solve it. The host runs a couple of seconds and then connection is lost. I have to kill the process to get it stopped.

I also see that my Coral mini-PCIe card, which I'm passing through the VM does not work anymore in the VM.

apex ram did not enable within timeout (12000ms)

agent: 1
balloon: 0
bios: ovmf
boot: order=sata0
cores: 1
cpu: kvm64
efidisk0: SSD1TB:106/vm-106-disk-0.qcow2,size=128K
hostpci0: 0000:02:00.0
memory: 5120
name: haosova-6.6
net0: virtio=32:E3:A8:91:63:DE,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
sata0: SSD1TB:106/vm-106-disk-1.qcow2,size=160G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=0dfba893-733f-4db4-b386-337a0e9f63a4
sockets: 3
usb1: host=0451:bef3
vga: virtio
vmgenid: 2b7a8f06-195d-4789-838b-ad7cb2d64070

root@pve:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-5 (running version: 7.1-5/6fe299a0)
pve-kernel-5.13: 7.1-4
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-1-pve: 5.13.19-2
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.14-1
proxmox-backup-file-restore: 2.0.14-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.4-2
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-1
pve-qemu-kvm: 6.1.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-3
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

 

[t.lamprecht]

I've got the same problem with one of my VM's, after upgrading to 7.1:

So you see exactly following parts of the authors originally posted error in the PVE journal/syslog?

KVM internal error. Suberror: 1
emulation failure
...

Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

 

[Polyphemus]

So you see exactly following parts of the authors originally posted error in the PVE journal/syslog?

KVM internal error. Suberror: 1
emulation failure
...

Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

Hmm, no, my apoligies for hijacking this topic then

I was triggered by the 'upgrading pve-qemu-kvm' and the command failed in syslog...

Shall I open a new topic?

 

[t.lamprecht]

Shall I open a new topic?

Yeah, as yours is different and OPs is solved that would be sensible.

 

[djgraff209]

Great to hear that, thanks for reporting back!

I'd think the new 5.13 kernel was the remedy here, at least if you rebooted after the update.
Out of interest, which CPU model is this host using?

The host system is a Dell R710 with two Xeon X5550 CPUs. And yes - after seeing the kernel update in the upgrade list I rebooted the system.

 

[stuckoff]

Hi
I have the same issue with Win10Prо installed on pve-7.2-7:

Aug 10 04:33:32 pve-01 QEMU[1837122]: KVM: entry failed, hardware error 0x80000021
Aug 10 04:33:32 pve-01 QEMU[1837122]: If you're running a guest on an Intel machine without unrestricted mode
Aug 10 04:33:32 pve-01 QEMU[1837122]: support, the failure can be most likely due to the guest entering an invalid
Aug 10 04:33:32 pve-01 QEMU[1837122]: state for Intel VT. For example, the guest maybe running in big real mode
Aug 10 04:33:32 pve-01 QEMU[1837122]: which is not supported on less recent Intel processors.
Aug 10 04:33:32 pve-01 QEMU[1837122]: EAX=00000000 EBX=0085000f ECX=00000000 EDX=7db32948
Aug 10 04:33:32 pve-01 QEMU[1837122]: ESI=1b8f3a60 EDI=0085000f EBP=6e2dfec0 ESP=1fe7cfb0
Aug 10 04:33:32 pve-01 QEMU[1837122]: EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
Aug 10 04:33:32 pve-01 QEMU[1837122]: ES =0000 00000000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: CS =c200 7ffc2000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: SS =0000 00000000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: DS =0000 00000000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: FS =0000 00000000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: GS =0000 00000000 ffffffff 00809300
Aug 10 04:33:32 pve-01 QEMU[1837122]: LDT=0000 00000000 000fffff 00000000
Aug 10 04:33:32 pve-01 QEMU[1837122]: TR =0040 1fe63000 00000067 00008b00
Aug 10 04:33:32 pve-01 QEMU[1837122]: GDT=     1fe64fb0 00000057
Aug 10 04:33:32 pve-01 QEMU[1837122]: IDT=     00000000 00000000
Aug 10 04:33:32 pve-01 QEMU[1837122]: CR0=00050032 CR2=00ad03c0 CR3=74e4a000 CR4=00000000
Aug 10 04:33:32 pve-01 QEMU[1837122]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Aug 10 04:33:32 pve-01 QEMU[1837122]: DR6=00000000ffff4ff0 DR7=0000000000000400
Aug 10 04:33:32 pve-01 QEMU[1837122]: EFER=0000000000000000
Aug 10 04:33:32 pve-01 QEMU[1837122]: Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

I tried CPU Type: kvm64/Host .. no success.
Is there any working solutions/workarounds

Regards,
Stoyan

 

[Stoiko Ivanov]

I tried CPU Type: kvm64/Host .. no success.
Is there any working solutions/workarounds

If you're running PVE 7.2 - look at this thread:

https://forum.proxmox.com/threads/v...-hardware-error-0x80000021.109410/post-487228
short answer - try the kernel on pve-test - else set the tdp_mmu paramter of the kvm module to false

I hope this helps!