AMD Nested Virtualization not working after 7.1 upgrade

[0rb3]

I have a Windows 10 VM that was working properly with hyper-v (amd nested virtualization enabled) in a AMD Threadripper 2970wx with CPU host,hidden=1, however after upgrade to 7.1 the machine goes to 100% CPU right after starting it and it crashes after sometime and I get below error message in syslog per core assigned.

[Sat Nov 20 08:38:12 2021] SVM: kvm [4031]: vcpu0, guest rIP: 0xfffff86fca8f4e99 unimplemented wrmsr: 0xc0010115 data 0x0

root@ripper:~# pveversion
pve-manager/7.1-5/6fe299a0 (running kernel: 5.13.19-1-pve)
root@ripper:~# cat /sys/module/kvm_amd/parameters/nested
1

 

[0rb3]

Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.

 

[mmenaz]

Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.

IS NOT SOLVED, is a bug / problem of 5.13 kernel, soon or later 5.11 will be removed from Proxmox and these problems need to be analyzed and solved in the new kernel. Your "solution" is just a temporary workaround of a partial upgrade (running older kernel)

 

[0rb3]

I agree, it is not a solution but rather a temporal work around, however it seems this is a issue that was mitigated in newest kernels, I will have to wait for a PVE Kernel update to test if this issue was fixed.

 

[grusim]

Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.

just in case you didn't catch it yet, there's now an opt-in kernel package for pve-kernel-5.15 available...
I just installed it and for me it solved the issue on a AMD EPYC 3000 series CPU with WSL2 on a Win11 VM.

see here: https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/

 

[Quakes]

Hey Guys. What kernel have you had success on?

I have a Threadripper 3990x and as soon as I enable WSL2/Hyper-V the VM becomes unstable with the "unimplemented wrmsr" error.

Feel like i've tried about 20 kernels at this point.

Cheers

Jon

 

[wickedaverage]

I have a Ryzen 3950x and am having the same error after trying to enable Hyper-V. I've tried a few different kernels with the same results. Any suggestions?

[ 1909.038357] SVM: kvm [8179]: vcpu0, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.359347] SVM: kvm [8179]: vcpu1, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.481099] SVM: kvm [8179]: vcpu2, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.602830] SVM: kvm [8179]: vcpu3, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.724574] SVM: kvm [8179]: vcpu4, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.846336] SVM: kvm [8179]: vcpu5, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.968114] SVM: kvm [8179]: vcpu6, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1910.089876] SVM: kvm [8179]: vcpu7, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0

 

[matrix303]

So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.

 

[wickedaverage]

So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.

I think the nested virtualization issue is not related to GPU passthrough. (I am also ultimately going to pass through a GPU to my VM and was hoping to use Hyper-V's implementation of GPU-P).

But to simplify the problem I created another VM with no GPU passthrough and the same problem occurs with the VM freezing ONLY when Hyper-V is enabled.

The 'unimplemented wrmsr' error, while I do see it, still allows for GPU passthrough, no issues with stability. (This may be a red herring for you as it was for me before as well)

 

[AllanM]

Going to give this thread a nudge for some increased attention.

6 node cluster of matching 2113S-WTRT w/7402P CPU's.

We are attempting to enable credential guard / virtualization based security for our windows server guests. This is dependent on Hyper-V working on the guest. I believe the issues are related to those discussed in this thread.

At first, the guests were hard-locking on boot when we attempted to enable CG/VBS. I then enabled SR-IOV in BIOS (SVM was already enabled). Now the guest doesn't really "lock" but "spins" the dots forever while the host server terminal produces the "unimplemented wrmsr" error.

I wonder if this is a known issue and whether it is being addressed somewhere. Likely a kernel bug.

 

[AllanM]

7.2 Update seems to have resolved this issue for us. Have a guest now running CG/VBS on our AMD-EPYC-ROME cluster.

 

[wickedaverage]

7.2 Update seems to have resolved this issue for us. Have a guest now running CG/VBS on our AMD-EPYC-ROME cluster.

Great to hear! Any special configs on your L1 VM (to get Hyper-V working) ?

 

[AllanM]

Great to hear! Any special configs on your L1 VM (to get Hyper-V working) ?

Just set the hardware CPU type to "host" , seems to be working. Just migrated that running VM to another node on the cluster and it seems to be working even through a migration.

We also configured the virtual machine with QEMU 6.1, UEFI, secure boot, a UFI disk, and a vTPM.

Now... since we're just using virtualization based security, not *all* of hyper-v functionality, things may be a bit different for us. When VBS/credential guard is enabled in group policy on windows, it automatically installs a subset of Hyper-V features needed for this.