AMD Nested virtualization fails

[DougW]

I'm trying to get nested virtualization (Proxmox=>Win10=>Virtualbox=>Linux) working on a Ryden 3900.
I've followed the guide at https://pve.proxmox.com/wiki/Nested_Virtualization I've read the forums such as https://forum.proxmox.com/threads/windows-10-1809-nested-virtualization-does-not-work.52554/ but it still doesn't work. Currently, I have the VM set as:

agent: 1
args: -cpu 'host,+svm,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off,hypervisor=off'
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host,hidden=1,flags=+virt-ssbd;+amd-ssbd
efidisk0: rpool:vm-113-disk-1,size=1M
hostpci0: 2b:00,pcie=1
ide0: local:iso/virtio-win.iso,media=cdrom,size=363020K
machine: q35
memory: 32768
name: t-pain
net0: virtio=D6:13:96:FB:4A:7B,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
scsi0: rpool:vm-113-disk-0,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=5f58ad26-c659-4542-8fe4-536d73cd0323
sockets: 1
vga: none
vmgenid: 51e24c5f-5fa2-4a4f-a625-60b31dcdd633

runs fine and GPU passthrough on it is working. However, when I go to start any Virtualbox Linux VM inside it, the kernel locks up during the boot process. I suspect that the AMD path is less documented than the Intel one and I'm missing something simple. Has anyone gotten nested virtualization to work on a Ryzen with Proxmox + Win10 + Virtualbox? Or any ideas how to troubleshoot this?

 

[Stoiko Ivanov]

have you checked the nested virtualization page on our wiki?
https://pve.proxmox.com/wiki/Nested_Virtualization

What does:

cat /sys/module/kvm_amd/parameters/nested

show?

also virtualbox should work without virtualization (though very slow) and not lead to a kernel-panic or lockup
what does the nested VM show on the screen?

I hope this helps!

 

[DougW]

have you checked the nested virtualization page on our wiki?
https://pve.proxmox.com/wiki/Nested_Virtualization

What does:

cat /sys/module/kvm_amd/parameters/nested

show?

also virtualbox should work without virtualization (though very slow) and not lead to a kernel-panic or lockup
what does the nested VM show on the screen?

Hi, Thanks for your reply, as I said in the very first line I did indeed go through that page, as well as the forums in particular the thread I also linked. Nested virtualization does appear to be on in the kernel:

# cat /sys/module/kvm_amd/parameters/nested
1

I guess I should have better read the actual error:


If you recall I had
flags=+virt-ssbd;+amd-ssbd
set, so after reading this I tried turning it off with:
flags=-virt-ssbd;-amd-ssbd

but the same error is still there. I may want to update the topic to be Nested Virtualization Spectre Mitigation on AMD...

 

[Stoiko Ivanov]

Hi, Thanks for your reply, as I said in the very first line I did indeed go through that page, as well as the forums in particular the thread I also linked. Nested virtualization does appear to be on in the kernel:

sorry - should have read your post more carfully!

nesting is enabled on PVE.

The messages from the Virtualbox guest is not an error - just a notification, which mitigation for spectre/meldown/... is applied
that should not prevent the machine from booting.
maybe try to make the vm be more verbose while booting (e.g. remove the 'quiet' flag from the guests kernel command line - or whatever else is available for the guest os you try to run in virtualbox)

I hope this helps!

 

[DougW]

sorry - should have read your post more carfully!

nesting is enabled on PVE.

The messages from the Virtualbox guest is not an error - just a notification, which mitigation for spectre/meldown/... is applied
that should not prevent the machine from booting.
maybe try to make the vm be more verbose while booting (e.g. remove the 'quiet' flag from the guests kernel command line - or whatever else is available for the guest os you try to run in virtualbox)

I hope this helps!

Thank you for your fast replies and the context. That really helps. Removing the quiet flag does not help much, it's around USB initialization:


but I'm pretty sure the USB is a red herring.

 

[Stoiko Ivanov]

hmm - does the virtualbox start on a baremetal machine ? (i.e. on your laptop, or some other computer you have)

 

[DougW]

hmm - does the virtualbox start on a baremetal machine ? (i.e. on your laptop, or some other computer you have)

I've tried a few different iso's and a few different VM solutions. The end goal is getting an Android Emulator working like Bluestacks or Memu, (both of which fail to start their VMs. And unfortunately they don't show a console log to see where they fail.) Since I know Bluestacks is built on Virtualbox, I tried two different ISOs the one failing above is the Official Ubuntu 18.04 Desktop livecd with shasum matching what's on the website. Using the same ISO I can create a new VM in proxmox and get it to start, and can also get it to boot on my desktop. Also the other one I looked at Memu, is built with Hyper-V and it too fails to fully boot it's VM.


I think stepping back my question is has anyone successfully done this on a Ryzen 3900x and what was the configuration of the kvm kernel module and the arguments to qemu?

 

[tom]

Based on my knowledge, nested virtulaization with Windows OS does not work on AMD ryzen cpus. I finally moved to an Intel CPU for these workloads.

 

[onepamopa]

@tom any idea when this is gonna be fixed?

 

[Alwin]

@tom any idea when this is gonna be fixed?

Never. Or at least as long as Microsoft doesn't add support.
https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

 

[hurenkam]

Never. Or at least as long as Microsoft doesn't add support.
https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

There is still hope:
https://techcommunity.microsoft.com/t5/virtualization/amd-nested-virtualization-support/ba-p/1434841

 

[mcdull]

Today I also tried to install memu on my Win10 guest. Failed.
After managing to add +svm flag to conf, I tried installing vmware workstation under win10 guest, and succeed.
Then install another win10 inside the vmware workstation (with like 16 cores + 16GB ram)
And try memu inside that win10 guest (i.e. Proxmox --> win10--> vmware workstation --> win10 --> MEMU)
Same fail at 59%.

So does it mean nested virtalization is supported in windows but with limitation?