Tricki problem with Windows and nested Virtualization.

[heq]

Hi, I need to run very specific machine and I currently have some issues. There is even possibilty?

The main view: Proxmox -> Windows -> VMware Player -> Windows(VM).

Current configuration of VM


On the Windows I have a problem with the VMware Player when try to open VM.


There is no Hyper-V roles on that Windows. There is also the problem with disabling Memory Integrity.

I hope that the problems goes from Proxmox, cause when I use Windows directly install on Disk and use VMware Player with Windows VM all works fine.

 

[Moayad]

Hi,

Have you enabled the Nested in your Proxmox VE [0]? If yes - I would try with the best practices [1]

[0] https://pve.proxmox.com/wiki/Nested_Virtualization
[1] https://pve.proxmox.com/wiki/Windows_10_guest_best_practices

 

[aaron]

There is no Hyper-V roles on that Windows. There is also the problem with disabling Memory Integrity.

Is there also no WSL installed? That would pull in Hyper-V AFAIK.

 

[heq]

Hi,

Have you enabled the Nested in your Proxmox VE [0]? If yes - I would try with the best practices [1]

[0] https://pve.proxmox.com/wiki/Nested_Virtualization
[1] https://pve.proxmox.com/wiki/Windows_10_guest_best_practices

Enabled. Needed for another things, so I enable long time ago.

Is there also no WSL installed? That would pull in Hyper-V AFAIK.

No. There is no WSL installed. HyperV also not installed.

 

[p.b]

Enabled. Needed for another things, so I enable long time ago.

No. There is no WSL installed. HyperV also not installed.

If you are running a Windows VM in Proxmox, several default hv-* flags will be added automatically.
And you are seemingly using both a Windows VM with the host CPU type and even some additional hv-* flags!

You can check all effectively used hv-* flags by runnning qm showcmd 104 | grep --color "hv_" It'll highlight something like hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,hv_tlbflush,hv_evmcs,...

Now according to the documentation of Hyper-V Enlightenments, "These features make Windows and Hyper-V guests think they’re running on top of a Hyper-V compatible hypervisor and use Hyper-V specific features."

I have no experience with VMWare (Player) but you could instead try to remove all your manually added CPU flags (hv_tlbflush, hv_evmcs and the redundant aes flag) and additionally get rid of all silently added default enlightenments by using some custom args line in your VM config (/etc/pve/qemu-server/104.conf) like below:

args: -cpu host,-hv_ipi,-hv_relaxed,-hv_reset,-hv_runtime,-hv_spinlocks,-hv_stimer,-hv_synic,-hv_time,-hv_vapic,-hv_vpindex

Afterwards, there is a PowerShell tool from Microsoft which provides some easy way of making sure that any traces from Hyper-V are disabled during boot (needs to be run from an elevated PowerShell prompt):

DG_Readiness.ps1 –Disable -AutoReboot

But even if you manage to get rid of Hyper-V completely, VMWare still needs to be compatible with nesting in KVM in the end. Are you really sure about that? You will also (very likely) get very poor performance in your L1 Windows VM and even worse in the nested L2 VMs in VMWare Player.

I'd rather recommend to get at least rid of VMWare Player by replacing it with Hyper-V for the nested L2 VMs. Converting VMWare stuff to Hyper-V seems to be possible. If you use all available hv-* flags in Proxmox for Hyper-V in Windows, performance on both L1 and L2 will probably be much better.

Of course the optimal solution would be to run the nested VMs directly on Proxmox instead.