Nested - VMWare Workstation inside Windows 10 Proxmox Guest - can’t start VMWare guests

[tenly]

Physical Hardware:
- Mini-PC from MSECORE with
- Core i9-9880H / 32GB RAM
- 1TB NVME SSD, 4TB NVME SSD
- VTX enabled in BIOS

Proxmox v7.3 installed on 1TB NVME SSD, Storage Pool on 4TB NVME SSD

Proxmox Guest:
- Windows 10 Enterprise 20H2 (fresh install)
- (Have also tried with 22H2)
- 4 Cores / 16GB RAM
- Windows Updates to March 2023
- Hyper-V completely disabled, Credential Guard, Windows Sandbox, Core Isolation - all disabled

- Installed VMWare Workstation Pro 15.5.7
- (have also tried with current version 17)
- copied working CentOS guest vm from another machine
- (have also tried with New Ubuntu VM)
- VMWare Workstation starts without any problems, but when attempting to power on any VM, a popup appears stating:
"VMware Workstation and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware Workstation."

- I am very confident that Hyper-V and all of the windows components that secretly use Hyper-V technology have been disabled
- but even if they aren’t, Hyper-V and VMWare are supposed to be compatible now (since 2020)

I’m not sure I’ve done everything necessary in ProxMox to enable nested virtualization. I couldn’t find anything applicable to v7.x

On the Windows 10 guest, my config file shows:

agent: 1
cores: 4
cpu: host
numa: 1
memory: 16384
machine: pc-q35-7.2
balloon: 4096
boot: order=ide2;net0
meta: creation-qemu=7.2.0,ctime=1679429073
name: Win10-VMWare
net0: e1000=4A:74:9B:61:71:EE,bridge=vmbr0,queues=1
ostype: win10
parent: Freshly_Configured_Win1020h2
sata0: local-4tb-images:100/base-100-disk-1.qcow2/101/vm-101-disk-0.qcow2,size=768G
scsihw: virtio-scsi-single
smbios1: uuid=c0c54e5b-fe5e-496f-8ec7-d5d97896e512
sockets: 1
vga: vmware,memory=256
vmgenid: ac35cd9a-e959-44f2-9d1a-eb2eed79dbbb
ide2:
iso_extnvme2tb:iso/en_windows_10_business_editions_version_20h2_updated_april_2021_x64_dvd_61562f02.iso,media=cdrom,size=6058782K

- Do I need additional flags on the cpu line?

And in the guest options, I have:
KVM Hardware Virtualization - enabled

Is there anything in the Proxmox general configuration that I need to check or enable?

Note: During most of my testing, over the past 2 days, this Windows 10 guest was the only guest VM running in Proxmox.

I will greatly appreciate any responses that will help me solve this or troubleshoot the problem further.

 

[klaas.sysop]

Physical Hardware:
- Mini-PC from MSECORE with
- Core i9-9880H / 32GB RAM
- 1TB NVME SSD, 4TB NVME SSD
- VTX enabled in BIOS

Proxmox v7.3 installed on 1TB NVME SSD, Storage Pool on 4TB NVME SSD

Proxmox Guest:
- Windows 10 Enterprise 20H2 (fresh install)
- (Have also tried with 22H2)
- 4 Cores / 16GB RAM
- Windows Updates to March 2023
- Hyper-V completely disabled, Credential Guard, Windows Sandbox, Core Isolation - all disabled

- Installed VMWare Workstation Pro 15.5.7
- (have also tried with current version 17)
- copied working CentOS guest vm from another machine
- (have also tried with New Ubuntu VM)
- VMWare Workstation starts without any problems, but when attempting to power on any VM, a popup appears stating:
"VMware Workstation and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware Workstation."

- I am very confident that Hyper-V and all of the windows components that secretly use Hyper-V technology have been disabled
- but even if they aren’t, Hyper-V and VMWare are supposed to be compatible now (since 2020)

I’m not sure I’ve done everything necessary in ProxMox to enable nested virtualization. I couldn’t find anything applicable to v7.x

On the Windows 10 guest, my config file shows:

agent: 1
cores: 4
cpu: host
numa: 1
memory: 16384
machine: pc-q35-7.2
balloon: 4096
boot: order=ide2;net0
meta: creation-qemu=7.2.0,ctime=1679429073
name: Win10-VMWare
net0: e1000=4A:74:9B:61:71:EE,bridge=vmbr0,queues=1
ostype: win10
parent: Freshly_Configured_Win1020h2
sata0: local-4tb-images:100/base-100-disk-1.qcow2/101/vm-101-disk-0.qcow2,size=768G
scsihw: virtio-scsi-single
smbios1: uuid=c0c54e5b-fe5e-496f-8ec7-d5d97896e512
sockets: 1
vga: vmware,memory=256
vmgenid: ac35cd9a-e959-44f2-9d1a-eb2eed79dbbb
ide2:
iso_extnvme2tb:iso/en_windows_10_business_editions_version_20h2_updated_april_2021_x64_dvd_61562f02.iso,media=cdrom,size=6058782K

- Do I need additional flags on the cpu line?

And in the guest options, I have:
KVM Hardware Virtualization - enabled

Is there anything in the Proxmox general configuration that I need to check or enable?

Note: During most of my testing, over the past 2 days, this Windows 10 guest was the only guest VM running in Proxmox.

I will greatly appreciate any responses that will help me solve this or troubleshoot the problem further.

Did you solve it? i want to do the same thing

 

[lauhub]

I finally manage to make my VMware run inside a Win11 installation on PVE.

I had to add the following in the VM's `/etc/pve/qemu-server/<VMID>.conf`:

1. remove the `cpu:` line
2. add a `args` line as follow:

```
args: -cpu host,+vmx
```

This worked for me, but I wanted to get more capabilities from my CPU. So, actually, I got help from a AI chatbot which analyzed the output from `/proc/cpuinfo` on my PVE. `grep vmx /proc/cpuinfo` ouput helped it to create the arguments set below:

```
args: -cpu host,+vmx,+ssse3,+sse4.1,+sse4.2,+aes,+xsave,+xsaveopt,+xsavec,+xgetbv1,+xsaves,+avx,+avx2,+fma,+bmi1,+bmi2,+invtsc,+pdpe1gb,+tsc-deadline,+hypervisor
```

If needed, try without `+hypervisor` (but I didn't have to).

Below is the memo I asked it to help creating this arguments set.
⸻
General walkthrough

1. Inspect the host CPU capabilities
On the Proxmox host:

```
cat /proc/cpuinfo
```

Look at the two sections:
• flags → general CPU instruction set features.
• vmx flags (Intel) or svm flags (AMD) → virtualization-specific capabilities.

Example from the host (Intel):


```
flags : ... ssse3 sse4_1 sse4_2 aes xsave avx avx2 fma bmi1 bmi2 ...
vmx flags : ... ept vpid unrestricted_guest ...
```

⸻
2. Identify features relevant for nested virtualization

From vmx flags pick the virtualization extensions required by VMware/VirtualBox:
• +vmx → base Intel VT-x.
• +ept, +unrestricted_guest, +vpid → needed for running 64-bit nested guests.

From flags pick common instruction set extensions that modern hypervisors expect:
• SIMD: +ssse3, +sse4.1, +sse4.2
• Crypto: +aes
• Vector ops: +avx, +avx2, +fma
• State save/restore: +xsave, +xsaveopt, +xsavec, +xsaves, +xgetbv1
• Performance: +bmi1, +bmi2, +invtsc, +pdpe1gb, +tsc-deadline
⸻
3. Translate into QEMU arguments

Proxmox’s cpu: directive does not always accept adding +vmx.
Instead, use the args: directive to override QEMU’s -cpu line:
args: -cpu host,+vmx,+ssse3,+sse4.1,+sse4.2,+aes,+xsave,+xsaveopt,+xsavec,+xgetbv1,+xsaves,+avx,+avx2,+fma,+bmi1,+bmi2,+invtsc,+pdpe1gb,+tsc-deadline
This ensures the guest OS and nested hypervisors see the same set of CPU capabilities that exist on the host.
⸻
4. Verify inside the guest

Inside the Windows or Linux guest, confirm the flags are visible:
• On Linux:
cat /proc/cpuinfo | grep -E "vmx|avx|fma|sse4"

• On Windows: use Coreinfo.exe (Sysinternals) or CPU-Z to check feature availability.
If vmx (Intel VT-x) or svm (AMD-V) is exposed, nested hypervisors like VMware Workstation can run.