Nested virtualization hyper-v amd not working

lukas12342

Member
May 25, 2018
7
0
6
I am able to use hyper-v on an my intel/kvm system but with my amd system the vm (server 2019) gets unresponsive and eventually crashes with a bluescreen.
Bluescreens seem to be all watchdog related but vary. (DPC Watchdog Violation,clock watchdog timeout).

Hardware platform:
CPU: AMD Ryzen Threadripper 2950X
Board: ASRock X399 Taichi X399-A (SVM, IOMMU related settings enabled in BIOS)


root@pve-master:~# grep -H '' /sys/module/kvm_amd/parameters/*
/sys/module/kvm_amd/parameters/avic:0
/sys/module/kvm_amd/parameters/nested:1
/sys/module/kvm_amd/parameters/npt:1
/sys/module/kvm_amd/parameters/pause_filter_count:3000
/sys/module/kvm_amd/parameters/pause_filter_count_grow:2
/sys/module/kvm_amd/parameters/pause_filter_count_max:65535
/sys/module/kvm_amd/parameters/pause_filter_count_shrink:0
/sys/module/kvm_amd/parameters/pause_filter_thresh:128
/sys/module/kvm_amd/parameters/sev:0
/sys/module/kvm_amd/parameters/vgif:1
/sys/module/kvm_amd/parameters/vls:1

root@pve-master:~# grep -H '' /sys/module/kvm/parameters/*
/sys/module/kvm/parameters/enable_vmware_backdoor:N
/sys/module/kvm/parameters/force_emulation_prefix:N
/sys/module/kvm/parameters/halt_poll_ns:200000
/sys/module/kvm/parameters/halt_poll_ns_grow:0
/sys/module/kvm/parametQers/halt_poll_ns_shrink:0
/sys/module/kvm/parameters/ignore_msrs:Y
/sys/module/kvm/parameters/kvmclock_periodic_sync:Y
/sys/module/kvm/parameters/lapic_timer_advance_ns:4294967295
/sys/module/kvm/parameters/min_timer_period_us:200
/sys/module/kvm/parameters/report_ignored_msrs:N
/sys/module/kvm/parameters/tsc_tolerance_ppm:250
/sys/module/kvm/parameters/vector_hashing:Y


/usr/bin/kvm -id 114 -name server2019 -chardev socket,id=qmp,path=/var/run/qemu-server/114.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/114.pid -daemonize -smbios type=1,uuid=8167be1e-8f3d-4667-8535-62a035ebd087 -smp 6,sockets=1,cores=6,maxcpus=6 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/114.vnc,password -no-hpet -cpu EPYC,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_tlbflush,hv_ipi,enforce,vendor=AuthenticAMD -m 8192 -device vmgenid,guid=53ded4c7-f067-4ae6-8ab0-6a571ccbc1ba -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device qxl-vga,id=vga,bus=pcie.0,addr=0x1 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:6ff4f776953c -drive file=/pool_t1/t1_dir//template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive file=/pool_t1/t1_dir//template/iso/winserver19_17763.1.180914-1434.rs5_release_SERVER_EVAL_x64fre_de-de.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201 -drive file=/dev/zvol/pool_t1/vmdata/vm-114-disk-0,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap114i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=16:72:12:9F:33:FA,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard -cpu host,+svm


Possible upstream bug ?
Bug 203477 - [AMD][KVM] Windows L1 guest becomes extremely slow and unusable after enabling Hyper-V
 
Nested Hyper-V cannot work on AMD, see https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

Are you sure you run exactly this on ESXi, I have my doubts.


Yes, ESXi can.

Here's a screenshot of the working environment.

I modified the esxi config file.

for gpu passthrough
```
hypervisor.cpuid.v0 = "FALSE"
vhv.allowPassthru = "TRUE"
svga.present = "FALSE"
```

for nested
```
vhv.enable = "TRUE"
```

When I add that setting to esxi it shows a warning, but changing the configuration file works fine.

sorry for my poor english.
 

Attachments

  • 1.jpg
    1.jpg
    323.4 KB · Views: 33
Which Windows 10 Build number do your run exactly?

I tested two versions.

ver. 2004 (build. 19628.1) for wsl2
ver. 1909 (build. 18363.959) for vm & android emulator

Everything works fine.

In proxmox, I thought that the most appropriate setting-cpu flags...- was applied, but after hyper-v setting, a blue screen appears during booting.
 
Running into the same issue here with an AMD Ryzen 3970x and Proxmox 6.2, I can install Windows 10 fine with all the required cpu flags but once I enabled Hyper-V from windows it causes it to BSOD.
 
Last edited:
You can sort of workaround it if you can run the Guest with 1 vCPU. I've been trying to test out WSL2 and this is the only way I got it to work on Proxmox so far. Have tried all of the extra arg: cpu options. Always bluescreens with Watchdog timeouts with >1 vCPU core. Able to install hyper-v and seems stable with 1 vCPU. Wasn't 100% able to get a VM spun up though, as Hyper-v manager times out. But WSL2 runs in a VM so as mentioned it "kinda" works.

args: -cpu host,hypervisor=off
cores: 1

1596913883470.png
 
Last edited:
On Intel CPU work with line in .conf
Code:
cpu: host             
args: -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx
 
  • Like
Reactions: Snowiness7674

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!