KVM internal error. Suberror: 3

[Kesha]

Hi, everybody

after installing windows 2008 R2, the virtual machine freezes and writes the error "internal error"

When viewing the logs of the data center , I found these messages

Mar 18 23:18:19 contec QEMU[9153]: KVM internal error. Suberror: 3
Mar 18 23:18:19 contec QEMU[9153]: extra data[0]: 0x000000008000001f
Mar 18 23:18:19 contec QEMU[9153]: extra data[1]: 0x0000000000000020
Mar 18 23:18:19 contec QEMU[9153]: extra data[2]: 0x0000000000000581
Mar 18 23:18:19 contec QEMU[9153]: extra data[3]: 0x000000000000000c
Mar 18 23:18:19 contec QEMU[9153]: RAX=0000000000000000 RBX=fffffa801a33f2a0 RCX=0000000040000070 RDX=0000000000000000
Mar 18 23:18:19 contec QEMU[9153]: RSI=000000000000b3d5 RDI=fffffa801fd22700 RBP=fffff88009a46ab0 RSP=fffff88009a46a08
Mar 18 23:18:19 contec QEMU[9153]: R8 =fffffa8018ce9b00 R9 =0000000000000000 R10=fffff880052a3340 R11=0000000000000000
Mar 18 23:18:19 contec QEMU[9153]: R12=fffffa80205096b0 R13=0000000000000000 R14=fffffa801a0e4280 R15=fffffa8018627980
Mar 18 23:18:19 contec QEMU[9153]: RIP=fffff800016e222c RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
Mar 18 23:18:19 contec QEMU[9153]: ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Mar 18 23:18:19 contec QEMU[9153]: CS =0010 0000000000000000 00000000 00209b00 DPL=0 CS64 [-RA]
Mar 18 23:18:19 contec QEMU[9153]: SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
Mar 18 23:18:19 contec QEMU[9153]: DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Mar 18 23:18:19 contec QEMU[9153]: FS =0053 00000000fffae000 00003c00 0040f300 DPL=3 DS [-WA]
Mar 18 23:18:19 contec QEMU[9153]: GS =002b fffff80001808d00 ffffffff 00c0f300 DPL=3 DS [-WA]
Mar 18 23:18:19 contec QEMU[9153]: LDT=0000 0000000000000000 ffffffff 00c00000
Mar 18 23:18:19 contec QEMU[9153]: TR =0040 fffff800037d7080 00000067 00008b00 DPL=0 TSS64-busy
Mar 18 23:18:19 contec QEMU[9153]: GDT= fffff800037d6000 0000007f
Mar 18 23:18:19 contec QEMU[9153]: IDT= fffff800037d6080 00000fff
Mar 18 23:18:19 contec QEMU[9153]: CR0=80050031 CR2=fffff8a008f93000 CR3=00000005727bf000 CR4=000006f8
Mar 18 23:18:19 contec QEMU[9153]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Mar 18 23:18:19 contec QEMU[9153]: DR6=00000000ffff0ff0 DR7=0000000000000400
Mar 18 23:18:19 contec QEMU[9153]: EFER=0000000000000d01
Mar 18 23:18:19 contec QEMU[9153]: Code=25 a8 4b 00 00 b9 70 00 00 40 0f ba 32 00 72 06 33 c0 8b d0 <0f> 30 5a 58 59 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 90 cc cc>

Help me figure out this problem.

 

[fiona]

Hi,
please share the output of pveversion -v and qm config <ID> with the ID of your VM. What CPU model does your host have?

Does "after installing" mean directly after installing or during the boot afterwards? How far does the VM get during boot approximately?

 

[Kesha]

pveversion -v

Linux contec 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Mar 19 12:01:27 +09 2023 on pts/0
root@contec:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

qm config <ID>

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Mar 20 22:00:13 +09 2023 on pts/0
root@contec:~# qm config
400 not enough arguments
qm config <vmid> [OPTIONS]
root@contec:~# qm config 287
boot: order=ide0;ide2
cores: 4
cpu: kvm64
cpuunits: 9998
description: qmdump#map%3Aide1%3Adrive-ide1%3Assd2%3Araw%3A
ide0: ssdsystem:vm-287-disk-0,backup=0,size=400G
ide1: ssd2:vm-287-disk-0,size=180G
ide2: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
kvm: 1
machine: pc-i440fx-7.1
memory: 31476
net0: e1000=52:3F:58B:B9:09,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: win7
protection: 1
scsihw: virtio-scsi-pci
smbios1: uuid=720e3019-27f4-4f62-b57b-62006e90ba13
sockets: 4
vcpus: 16
vmgenid: f529fc3f-625f-4681-b0f9-104a30df36a2

 

[Kesha]

What CPU model does your host have?

Intel(R) Core(TM) i9-12900K

 

[Kesha]

After booting, the guest OS can work from 20 to 25 hours

 

[Kesha]

I'm leaning towards the broken iso image version

 

[fiona]

You could try updating the kernel to the latest 5.15 kernel or also try out the 6.2 opt-in kernel and see if the issue persists.

 

[Kesha]

Updated , let 's see what happens

 

[Kesha]

root@contec:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 6.2.6-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-6.2: 7.3-8
pve-kernel-helper: 7.3-8
pve-kernel-5.15: 7.3-3
pve-kernel-6.2.6-1-pve: 6.2.6-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-6
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20221111-1
pve-firewall: 4.2-7
pve-firmware: 3.6-4
pve-ha-manager: 3.5.1
pve-i18n: 2.8-3
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[Kesha]

You could try updating the kernel to the latest 5.15 kernel or also try out the 6.2 opt-in kernel and see if the issue persists.

It didn't help. The virtual machine has worked for 12 hours and again an internal error . What else can be done???

 

[fiona]

Unfortunately,

Mar 18 23:18:19 contec QEMU[9153]: KVM internal error. Suberror: 3
Mar 18 23:18:19 contec QEMU[9153]: extra data[0]: 0x000000008000001f
Mar 18 23:18:19 contec QEMU[9153]: extra data[1]: 0x0000000000000020

Looking at the kernel sources, if I'm interpreting things right (no guarantee there), because this is suberror 3, the 0x0...0020 = 32 here is the exit_reason.full being EXIT_REASON_MSR_WRITE.

Are you doing nested virtualization? If yes, please try the workaround suggested here. And even if not, you could still try it. Another thing you could try is using CPU type host.

 

[sergio.carvalho]

This error is certainly triggered by virtualization features in Windows. I tried disabling Hyper-V with no success. My alternative was to disable nested virtualization in the (linux) host. In my case, an intel machine, "modprobe -r kvm_intel", then "modprobe kvm_intel nested=0". Problem solved.

I later made the change permanent in /etc/modprobe.d.

The same result should be achievable disabling Hyper-V in the windows guest. I was surely missing something.

 

[Anatoliy221]

This error is certainly triggered by virtualization features in Windows. I tried disabling Hyper-V with no success. My alternative was to disable nested virtualization in the (linux) host. In my case, an intel machine, "modprobe -r kvm_intel", then "modprobe kvm_intel nested=0". Problem solved.

I later made the change permanent in /etc/modprobe.d.

The same result should be achievable disabling Hyper-V in the windows guest. I was surely missing something.

Good day. Tell me how to register the loading parameters of this module, what would it constantly load like that? Also faced with such a situation on core i9-14900

 

[Anatoliy221]

I asked , and answer myself:

echo "options kvm_intel nested = 0" >/ etc/modprobe.d/kvm_intel.conf
echo "kvm_intel nested=0" > /etc/modules-load.d/kvm_intel.conf
update-initramfs -u