Nested virtualization hyper-v amd not working

[lukas12342]

I am able to use hyper-v on an my intel/kvm system but with my amd system the vm (server 2019) gets unresponsive and eventually crashes with a bluescreen.
Bluescreens seem to be all watchdog related but vary. (DPC Watchdog Violation,clock watchdog timeout).

Hardware platform:
CPU: AMD Ryzen Threadripper 2950X
Board: ASRock X399 Taichi X399-A (SVM, IOMMU related settings enabled in BIOS)


root@pve-master:~# grep -H '' /sys/module/kvm_amd/parameters/*
/sys/module/kvm_amd/parameters/avic:0
/sys/module/kvm_amd/parameters/nested:1
/sys/module/kvm_amd/parameters/npt:1
/sys/module/kvm_amd/parameters/pause_filter_count:3000
/sys/module/kvm_amd/parameters/pause_filter_count_grow:2
/sys/module/kvm_amd/parameters/pause_filter_count_max:65535
/sys/module/kvm_amd/parameters/pause_filter_count_shrink:0
/sys/module/kvm_amd/parameters/pause_filter_thresh:128
/sys/module/kvm_amd/parameters/sev:0
/sys/module/kvm_amd/parameters/vgif:1
/sys/module/kvm_amd/parameters/vls:1

root@pve-master:~# grep -H '' /sys/module/kvm/parameters/*
/sys/module/kvm/parameters/enable_vmware_backdoor:N
/sys/module/kvm/parameters/force_emulation_prefix:N
/sys/module/kvm/parameters/halt_poll_ns:200000
/sys/module/kvm/parameters/halt_poll_ns_grow:0
/sys/module/kvm/parametQers/halt_poll_ns_shrink:0
/sys/module/kvm/parameters/ignore_msrs:Y
/sys/module/kvm/parameters/kvmclock_periodic_sync:Y
/sys/module/kvm/parameters/lapic_timer_advance_ns:4294967295
/sys/module/kvm/parameters/min_timer_period_us:200
/sys/module/kvm/parameters/report_ignored_msrs:N
/sys/module/kvm/parameters/tsc_tolerance_ppm:250
/sys/module/kvm/parameters/vector_hashing:Y


/usr/bin/kvm -id 114 -name server2019 -chardev socket,id=qmp,path=/var/run/qemu-server/114.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/114.pid -daemonize -smbios type=1,uuid=8167be1e-8f3d-4667-8535-62a035ebd087 -smp 6,sockets=1,cores=6,maxcpus=6 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/114.vnc,password -no-hpet -cpu EPYC,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,enforce,vendor=AuthenticAMD -m 8192 -device vmgenid,guid=53ded4c7-f067-4ae6-8ab0-6a571ccbc1ba -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:6ff4f776953c -drive file=/pool_t1/t1_dir//template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive file=/pool_t1/t1_dir//template/iso/winserver19_17763.1.180914-1434.rs5_release_SERVER_EVAL_x64fre_de-de.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201 -drive file=/dev/zvol/pool_t1/vmdata/vm-114-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap114i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=16:72:12:9F:33:FA,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard -cpu host,+svm


Possible upstream bug ?
Bug 203477 - [AMD][KVM] Windows L1 guest becomes extremely slow and unusable after enabling Hyper-V

 

[lukas12342]

This is certainly going to be annoying when wsl2 (requires hyper-v) reaches stable windows 10.

 

[Beu]

Hi, did you find a workaround ?

 

[latixia]

Same issue here. Is there a good solution? No problem with esxi.

 

[tom]

Same issue here. Is there a good solution? No problem with esxi.

Nested Hyper-V cannot work on AMD, see https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

Are you sure you run exactly this on ESXi, I have my doubts.

 

[latixia]

Nested Hyper-V cannot work on AMD, see https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

Are you sure you run exactly this on ESXi, I have my doubts.

Yes, ESXi can.

Here's a screenshot of the working environment.

I modified the esxi config file.

for gpu passthrough
```
hypervisor.cpuid.v0 = "FALSE"
vhv.allowPassthru = "TRUE"
svga.present = "FALSE"
```

for nested
```
vhv.enable = "TRUE"
```

When I add that setting to esxi it shows a warning, but changing the configuration file works fine.

sorry for my poor english.

 

[tom]

Yes, ESXi can.

Which Windows 10 Build number do your run exactly?

 

[latixia]

Which Windows 10 Build number do your run exactly?

I tested two versions.

ver. 2004 (build. 19628.1) for wsl2
ver. 1909 (build. 18363.959) for vm & android emulator

Everything works fine.

In proxmox, I thought that the most appropriate setting-cpu flags...- was applied, but after hyper-v setting, a blue screen appears during booting.

 

[uniacid]

Running into the same issue here with an AMD Ryzen 3970x and Proxmox 6.2, I can install Windows 10 fine with all the required cpu flags but once I enabled Hyper-V from windows it causes it to BSOD.

 

[netnem]

You can sort of workaround it if you can run the Guest with 1 vCPU. I've been trying to test out WSL2 and this is the only way I got it to work on Proxmox so far. Have tried all of the extra arg: cpu options. Always bluescreens with Watchdog timeouts with >1 vCPU core. Able to install hyper-v and seems stable with 1 vCPU. Wasn't 100% able to get a VM spun up though, as Hyper-v manager times out. But WSL2 runs in a VM so as mentioned it "kinda" works.

args: -cpu host,hypervisor=off
cores: 1

 

[tom]

Running into the same issue here with an AMD Ryzen 3970x and Proxmox 6.2, I can install Windows 10 fine with all the required cpu flags but once I enabled Hyper-V from windows it causes it to BSOD.

See https://pve.proxmox.com/wiki/Nested_Virtualization#Requirements

 

[Supme]

On Intel CPU work with line in .conf

cpu: host             
args: -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx