AMD Nested Virtualization not working after 7.1 upgrade

0

0rb3

New Member
Nov 20, 2021
3
0
1
35
I have a Windows 10 VM that was working properly with hyper-v (amd nested virtualization enabled) in a AMD Threadripper 2970wx with CPU host,hidden=1, however after upgrade to 7.1 the machine goes to 100% CPU right after starting it and it crashes after sometime and I get below error message in syslog per core assigned.

Code: 
[Sat Nov 20 08:38:12 2021] SVM: kvm [4031]: vcpu0, guest rIP: 0xfffff86fca8f4e99 unimplemented wrmsr: 0xc0010115 data 0x0

root@ripper:~# pveversion
pve-manager/7.1-5/6fe299a0 (running kernel: 5.13.19-1-pve)
root@ripper:~# cat /sys/module/kvm_amd/parameters/nested
1
 
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.
 
0rb3 said:
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.
Click to expand...
IS NOT SOLVED, is a bug / problem of 5.13 kernel, soon or later 5.11 will be removed from Proxmox and these problems need to be analyzed and solved in the new kernel. Your "solution" is just a temporary workaround of a partial upgrade (running older kernel)
 
I agree, it is not a solution but rather a temporal work around, however it seems this is a issue that was mitigated in newest kernels, I will have to wait for a PVE Kernel update to test if this issue was fixed.
 
0rb3 said:
Issue was the new kernel that comes with Proxmox 7.1 (5.13), I rebooted with previous Kernel 5.11 and everything is working fine without changing any configuration.

If any Mod see this , please mark this as SOLVED.
Click to expand...

just in case you didn't catch it yet, there's now an opt-in kernel package for pve-kernel-5.15 available...
I just installed it and for me it solved the issue on a AMD EPYC 3000 series CPU with WSL2 on a Win11 VM.

see here: https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/
 
Hey Guys. What kernel have you had success on?

I have a Threadripper 3990x and as soon as I enable WSL2/Hyper-V the VM becomes unstable with the "unimplemented wrmsr" error.

Feel like i've tried about 20 kernels at this point.

Cheers

Jon
 
I have a Ryzen 3950x and am having the same error after trying to enable Hyper-V. I've tried a few different kernels with the same results. Any suggestions?

[ 1909.038357] SVM: kvm [8179]: vcpu0, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.359347] SVM: kvm [8179]: vcpu1, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.481099] SVM: kvm [8179]: vcpu2, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.602830] SVM: kvm [8179]: vcpu3, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.724574] SVM: kvm [8179]: vcpu4, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.846336] SVM: kvm [8179]: vcpu5, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1909.968114] SVM: kvm [8179]: vcpu6, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0


[ 1910.089876] SVM: kvm [8179]: vcpu7, guest rIP: 0xfffff8323e7d68d7 unimplemented wrmsr: 0xc0010115 data 0x0
Click to expand...
 
So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.
 
Last edited:
matrix303 said:
So just to shed some light on this AMD nested virtualization issues:

TLDR: pve5.15 - HyperV and Sandbox tested on Windows VM WORKING -- BUT SAME VM, GPU passthrough and i get the "unimplemented wrmsr error"

Configuration:
- amd ryzen 5950x
- asus x570 wifi gaming ii

Started with pve5.13, I have a primary windows VM with GPU passthrough. I wanted to run hyperV (WSL and sandbox) on it.
- I started with the CPU=default (KVM64), but cant enable hyperV.
- changed to CPU=host, got into windows, enabled CPU. restarted, hung on boot... no luck
- upgraded pve to 5.15.30-1-pve
- Got the unimplemented wrmsr issue.

NEW windows VM saga:
- created NEW windows VM (no gpu passthrough), set CPU to host, and enabled hyperV and tested with sandbox... it works... note, i was using the pve console to interact with this new windows VM
- shutdown new windows VM, added my GPU... and ran into the unimplemented wrmsr issue.
- removed GPU, new windows VM booted and sandbox was working

Back to the old windows VM:
- removed the GPU, (hyperV enables, CPU=host still) -> using console, windows vm boots and sandbox works
- added GPU back... windows doesnt boot... black screen with cursor and then
- revert back to pve5.13 (same setup as original), GPU windows works, but no hyperV/Sandbox

Overall, I think that what is causing this sort of issue is actually gpu passthrough. Note GPU passthrough works flawlessly without hyperV/nested virtualization.

Hopefully someone with more knowledge can put two and two together and figure this out for us, more than happy to provide whatever info you need.
Click to expand...
I think the nested virtualization issue is not related to GPU passthrough. (I am also ultimately going to pass through a GPU to my VM and was hoping to use Hyper-V's implementation of GPU-P).

But to simplify the problem I created another VM with no GPU passthrough and the same problem occurs with the VM freezing ONLY when Hyper-V is enabled.

The 'unimplemented wrmsr' error, while I do see it, still allows for GPU passthrough, no issues with stability. (This may be a red herring for you as it was for me before as well)
 
Last edited:
Going to give this thread a nudge for some increased attention.

6 node cluster of matching 2113S-WTRT w/7402P CPU's.

We are attempting to enable credential guard / virtualization based security for our windows server guests. This is dependent on Hyper-V working on the guest. I believe the issues are related to those discussed in this thread.

At first, the guests were hard-locking on boot when we attempted to enable CG/VBS. I then enabled SR-IOV in BIOS (SVM was already enabled). Now the guest doesn't really "lock" but "spins" the dots forever while the host server terminal produces the "unimplemented wrmsr" error.

I wonder if this is a known issue and whether it is being addressed somewhere. Likely a kernel bug.
 
wickedaverage said:
Great to hear! Any special configs on your L1 VM (to get Hyper-V working) ?
Click to expand...

Just set the hardware CPU type to "host" , seems to be working. Just migrated that running VM to another node on the cluster and it seems to be working even through a migration.

We also configured the virtual machine with QEMU 6.1, UEFI, secure boot, a UFI disk, and a vTPM.

Now... since we're just using virtualization based security, not *all* of hyper-v functionality, things may be a bit different for us. When VBS/credential guard is enabled in group policy on windows, it automatically installs a subset of Hyper-V features needed for this.
 
Last edited:
You must log in or register to reply here.
Top Bottom