Nested virtualization crashes guest when it's being actively used

[tapiopve]

Hello,

I'd like to use nested virtualization with a Windows 11 guest running Oracle VM VirtualBox (latest version). I've set CPU type to "host" in the configuration of the Windows 11 VM and installed Oracle VM VirtualBox. Now, I'm trying to start a VM in VirtualBox, but after 2-3 minutes, the Windows 11 guest crashes.

Proxmox -> Windows 11 guest -> Oracle VM VirtualBox -> Whatever guest, doesn't matter

Only the Windows 11 guest is crashing, all the other VMs running on Proxmox are still running fine after Windows 11 crashes. The logs also doesn't have any useful information, except that I logged in and reset the VM (crash happened at 23:35):

# journalctl
Dec 04 23:29:13 tapiolite smartd[884]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Dec 04 23:36:20 tapiolite pvedaemon[64684]: <root@pam> successful auth for user 'root@pam'
Dec 04 23:36:29 tapiolite pvedaemon[68103]: <root@pam> successful auth for user 'root@pam'
Dec 04 23:36:35 tapiolite pvedaemon[68103]: VM 100 qmp command failed - VM 100 qmp command 'guest-ping' failed - got timeout
Dec 04 23:36:35 tapiolite pvedaemon[68103]: VM 100 qmp command failed - VM 100 qmp command 'guest-ping' failed - got timeout
Dec 04 23:36:39 tapiolite pvedaemon[68711]: <root@pam> starting task UPID:tapiolite:00018593:003221DE:61ABED77:qmreset:100:root@pam:
Dec 04 23:36:39 tapiolite pvedaemon[68711]: <root@pam> end task UPID:tapiolite:00018593:003221DE:61ABED77:qmreset:100:root@pam: OK

# dmesg --ctime
[Sat Dec  4 22:03:28 2021] CIFS: VFS: \\censored has not responded in 180 seconds. Reconnecting...
[Sat Dec  4 23:36:42 2021] usb 1-4: reset full-speed USB device number 3 using xhci_hcd
[Sat Dec  4 23:36:43 2021] usb 3-1: reset full-speed USB device number 2 using xhci_hcd
[Sat Dec  4 23:36:43 2021] usb 1-3: reset full-speed USB device number 2 using xhci_hcd
[Sat Dec  4 23:36:53 2021] usb 3-1: reset full-speed USB device number 2 using xhci_hcd
[Sat Dec  4 23:36:54 2021] usb 1-3: reset full-speed USB device number 2 using xhci_hcd
[Sat Dec  4 23:36:54 2021] usb 1-4: reset full-speed USB device number 3 using xhci_hcd

All of this only indicates that I've reset the VM, nothing about the crash. Latest updates are applied as of time of writing. Any idea about that problem?

Spoiler: CONFIG AND SYSTEM INFORMATION

# pveversion -v
root@tapiolite:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

# qm config 100
agent: 1
args: -cpu host,+svm
balloon: 4096
bios: ovmf
boot: order=ide0;ide2;net0
cores: 12
efidisk0: zincmelanterite:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:08:00,pcie=1,x-vga=1
ide0: zincmelanterite:vm-100-disk-0,discard=on,size=512G,ssd=1
ide2: none,media=cdrom
machine: pc-q35-6.1
memory: 16384
meta: creation-qemu=6.1.0,ctime=1636998149
name: titanium
net0: e1000=4A:B6:18:67:26:56,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=8c498060-1f0b-4008-8945-5d1180378d42
sockets: 1
tpmstate0: zincmelanterite:vm-100-disk-2,size=4M,version=v2.0
usb0: host=046d:0a9c,usb3=1
usb1: host=046d:c092,usb3=1
usb2: host=046d:c336,usb3=1
vga: none
vmgenid: 6f85d83c-94a3-4d90-b43f-2f0a86556e89

 

[tom]

What kind of physical hardware do you run? (CPU, Mainboard)

Please try with our upcoming 5.15 kernel.

> apt install pve-kernel-5.15

 

[Prothane]

Hello I'm having the same problem. I have Dell R740 2nd gen intel scalable processor. Proxmox 7 > Win 2k19 > virtualbox > Guest (Red Hat) went i try to boot the Red Hat VM the windows locks up and becomes unresponsive. i can stop windows then reboot the machine. i did try " apt install pve-kernel-5.15 " as well i did verify that i get " Y" running the check script " root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested " any help would be great thanks

as well i did try Proxmox > Windows 11 > turn on Virtual Machine Platform. reboot windows to enable and windows lockup at boot.

 

[debsque]

Hello I'm having the same problem. I have Dell R740 2nd gen intel scalable processor. Proxmox 7 > Win 2k19 > virtualbox > Guest (Red Hat) went i try to boot the Red Hat VM the windows locks up and becomes unresponsive. i can stop windows then reboot the machine. i did try " apt install pve-kernel-5.15 " as well i did verify that i get " Y" running the check script " root@proxmox:~# cat /sys/module/kvm_intel/parameters/nested " any help would be great thanks

as well i did try Proxmox > Windows 11 > turn on Virtual Machine Platform. reboot windows to enable and windows lockup at boot.

Did you find a solution? I have this same issue. After I turn on Virtual Machine Platform, the guest won't boot unless I switch to kvm64. CPU type host no longer works - windows tries auto repair and fails . I'm trying to run Windows Subsystem for Android.

 

[Prothane]

Sorry but im still stuck on this too. I was hope for some kernel updates to fix the problem. but two weeks ago I tried still freeze on boot (same have to switch KVM64 to boot, not host mode), hopefully with time the kernel engineers can fix this problem.