[SOLVED] Windows 10 (1809): nested virtualization does not work

[udotirol]

In order to run Docker for Windows, I'm trying to setup nested virtualization for a Windows 10 guest on our up to date 5.3-11 installation.

I've followed the wiki describing the required steps https://pve.proxmox.com/wiki/Nested_Virtualization and
I've searched both the net and the forum, but unfortunately I don't seem to get anywhere, Windows keeps telling me that

"Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed."

The host's kernel is setup correctly to support virtualization

$ cat /sys/module/kvm_intel/parameters/nested
Y

I've changed the CPU type to "host" and if I install a linux guest, I can see the processors having vmx capabilities:

$ fgrep vmx /proc/cpuinfo
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm ibrs ibpb kaiser tpr_shadow vnmi flexpriority ept vpid tsc_adjust xsaveopt arat
[...]

Additionally, as laid out in https://forum.proxmox.com/threads/nested-virtualization-not-working-anymore.49665/, I've also switched the machine type to "pc-i440fx-2.11", but to no avail as well.

Coreinfo tells me this about my Windows installation:

Anything else I could try?

 

[mac.linux.free]

...disable kvm acceleration...

 

[udotirol]

thanks for the idea, but turning off KVM hardware virtualization ends up with the VM taking like 10 minutes to display the login screen only and another 10 minutes to display the ordinary desktop, so that's unusable.

 

[udotirol]

I've since also tried to reinstall with as little traces of virtualization (no virtio drivers etc), but again to no avail.

The docs mention a "hidden" option for CPU

hidden=<boolean> (default = 0)
Do not identify as a KVM virtual machine.

... but setting it 1 doesn't change anything (except adding "kvm=off" to qemu's cpu switch), so I am quite lost now ...

 

[udotirol]

so finally I've found a solution to get nested Hyper-V virtualization up and running on a virtualized Windows 10 pro 1809 installation.

As it seems, one additional qemu cpu flag is required to hide the hypervisor from Windows:
* hypervisor: this needs to be turned off as a cpu flag, ie "-cpu host,hypervisor=off"

Unfortunately, PVE doesn't allow you to simply add those flags to the "cpu:" section in the vm config file, instead you have to (ab)use the "args:" line, so that my full vm config file now looks like this:

balloon: 0
boot: dcn
bootdisk: ide0
cores: 4
cpu: host
args: -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,-hypervisor
ide1: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
ide2: local:iso/Win10_1809Oct_English_x64.iso,media=cdrom,size=4956582K
memory: 4096
name: winslave1
net0: virtio=XX:XX:XX:XX:XX,bridge=vmbr3
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=xxx
sockets: 1
virtio0: iscsi-rambler-slowpool:vm-116-disk-0,cache=writeback,size=200G
vmgenid: xxx

The "args:" line you see here is the copied -cpu part from the original qemu commandline created by proxmox plus -hypervisor at the end (could also be written as hypervisor=off).

 

[dcsapak]

can you make a feature request ? https://bugzilla.proxmox.com/
we can probably add the hypervisor flag to the list of flags that the users can set so the resulting config would be then:

cpu: host,flags=-hypervisor

 

[udotirol]

I'll do that, yes.

But I'm still investigating if nested Hyper-V is usable for my workload, however.

Unfortunately, my tests have not been very promising in terms of performance so far, at least not when using it for Docker on Windows.

 

[seco]

It is very benifit for me,thank you.Also try the win19 can't boot anymore after install hyperv role in win19.

 

[gocloud]

so finally I've found a solution to get nested Hyper-V virtualization up and running on a virtualized Windows 10 pro 1809 installation.

As it seems, one additional qemu cpu flag is required to hide the hypervisor from Windows:
* hypervisor: this needs to be turned off as a cpu flag, ie "-cpu host,hypervisor=off"

Unfortunately, PVE doesn't allow you to simply add those flags to the "cpu:" section in the vm config file, instead you have to (ab)use the "args:" line, so that my full vm config file now looks like this:

balloon: 0
boot: dcn
bootdisk: ide0
cores: 4
cpu: host
args: -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,-hypervisor
ide1: local:iso/virtio-win-0.1.141.iso,media=cdrom,size=309208K
ide2: local:iso/Win10_1809Oct_English_x64.iso,media=cdrom,size=4956582K
memory: 4096
name: winslave1
net0: virtio=XX:XX:XX:XX:XX,bridge=vmbr3
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=xxx
sockets: 1
virtio0: iscsi-rambler-slowpool:vm-116-disk-0,cache=writeback,size=200G
vmgenid: xxx

The "args:" line you see here is the copied -cpu part from the original qemu commandline created by proxmox plus -hypervisor at the end (could also be written as hypervisor=off).

after adding the args, the vm performace is extremly slow

 

[zeldrix]

Thank you so muck

 

[mmhx]

Sadly this didn't work for me (tried with Windows Server 2022)
I have enabled nested virtualization, and adjusted the config to use the arguments from cpu out of
qm showcmd <ID> --pretty
in the args line as mentioned above (-hypervisor)

As soon as I install the hypervisor role the VM freezes/crashed during boot.

Has anybody managed to run this on Win11 or Win 2022 ?

 

[udotirol]

It is almost 3 years now that I posted my "solution" and things have evolved a lot, since. QEMU and the linux kernel have received numerous updates, that also affect nested virtualization.

Because of both performance and stability issues, we've since completely gotten rid of all Windows VMs providing (nested) virtualization and are instead operating them on bare metal for the sole purpose of functioning as a hypervisor for (docker style) Windows VMs.

Everything else just wasn't production ready, at least not for our workloads.

 

[majicf]

That's really sad to hear. Lab environments for docker development is getting more and more common. Not supporting nested virtualisation like other hypervisors can is a big problem