AMD Nested Virtualization not working after 7.1 upgrade

0rb3

New Member
Nov 20, 2021
3
0
1
34
I have a Windows 10 VM that was working properly with hyper-v (amd nested virtualization enabled) in a AMD Threadripper 2970wx with CPU host,hidden=1, however after upgrade to 7.1 the machine goes to 100% CPU right after starting it and it crashes after sometime and I get below error message in syslog per core assigned.

Code:
[Sat Nov 20 08:38:12 2021] SVM: kvm [4031]: vcpu0, guest rIP: 0xfffff86fca8f4e99 unimplemented wrmsr: 0xc0010115 data 0x0

root@ripper:~# pveversion
pve-manager/7.1-5/6fe299a0 (running kernel: 5.13.19-1-pve)
root@ripper:~# cat /sys/module/kvm_amd/parameters/nested
1
 
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.
 
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.
IS NOT SOLVED, is a bug / problem of 5.13 kernel, soon or later 5.11 will be removed from Proxmox and these problems need to be analyzed and solved in the new kernel. Your "solution" is just a temporary workaround of a partial upgrade (running older kernel)
 
I agree, it is not a solution but rather a temporal work around, however it seems this is a issue that was mitigated in newest kernels, I will have to wait for a PVE Kernel update to test if this issue was fixed.
 
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.

just in case you didn't catch it yet, there's now an opt-in kernel package for pve-kernel-5.15 available...
I just installed it and for me it solved the issue on a AMD EPYC 3000 series CPU with WSL2 on a Win11 VM.

see here: https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/
 
Hey Guys. What kernel have you had success on?

I have a Threadripper 3990x and as soon as I enable WSL2/Hyper-V the VM becomes unstable with the "unimplemented wrmsr" error.

Feel like i've tried about 20 kernels at this point.

Cheers

Jon
 
I have a Ryzen 3950x and am having the same error after trying to enable Hyper-V. I've tried a few different kernels with the same results. Any suggestions?

[ 1909.038357] SVM: kvm [8179]: vcpu0, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.359347] SVM: kvm [8179]: vcpu1, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.481099] SVM: kvm [8179]: vcpu2, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.602830] SVM: kvm [8179]: vcpu3, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.724574] SVM: kvm [8179]: vcpu4, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.846336] SVM: kvm [8179]: vcpu5, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.968114] SVM: kvm [8179]: vcpu6, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1910.089876] SVM: kvm [8179]: vcpu7, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0
 
So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.
 
Last edited:
So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.
I think the nested virtualization issue is not related to GPU passthrough. (I am also ultimately going to pass through a GPU to my VM and was hoping to use Hyper-V's implementation of GPU-P).

But to simplify the problem I created another VM with no GPU passthrough and the same problem occurs with the VM freezing ONLY when Hyper-V is enabled.

The 'unimplemented wrmsr' error, while I do see it, still allows for GPU passthrough, no issues with stability. (This may be a red herring for you as it was for me before as well)
 
Last edited:
Going to give this thread a nudge for some increased attention.

6 node cluster of matching 2113S-WTRT w/7402P CPU's.

We are attempting to enable credential guard / virtualization based security for our windows server guests. This is dependent on Hyper-V working on the guest. I believe the issues are related to those discussed in this thread.

At first, the guests were hard-locking on boot when we attempted to enable CG/VBS. I then enabled SR-IOV in BIOS (SVM was already enabled). Now the guest doesn't really "lock" but "spins" the dots forever while the host server terminal produces the "unimplemented wrmsr" error.

I wonder if this is a known issue and whether it is being addressed somewhere. Likely a kernel bug.
 
Great to hear! Any special configs on your L1 VM (to get Hyper-V working) ?

Just set the hardware CPU type to "host" , seems to be working. Just migrated that running VM to another node on the cluster and it seems to be working even through a migration.

We also configured the virtual machine with QEMU 6.1, UEFI, secure boot, a UFI disk, and a vTPM.

Now... since we're just using virtualization based security, not *all* of hyper-v functionality, things may be a bit different for us. When VBS/credential guard is enabled in group policy on windows, it automatically installs a subset of Hyper-V features needed for this.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!