Win11 VM opening many tabs at once crashes proxmox host

[jeenam]

This probably works because you are not using the type "Host" in this config.
I am still using `x86-64-v4` without any crashes since my last post.
But I still can't use Host type CPU. I mean I don't dare. Didn't retry since then.

Because of this
- I can't do any nested virtualization, like installing docker or hyper-v
- I can't passthrough a GPU

Can't comment on the nested virtualization issue as I haven't tried it.

Regarding docker, I have had zero problems running Linux VM's with CPU Type = host. Docker also runs fine on Linux VM's with CPU Type = host. Tested with multiple Debian 12 VM's running docker containers for 389-ds. The random crashes only occur when running Windows 11 VM's (haven't tried Windows 10 - though I suspect the same problem would occur).

There's obviously an issue with Zen 4 CPU's, KVM and virtualization of Windows 11 when using CPU type = host. I'd imagine it affects Windows Server as well, but have not yet tested it. It would seem to be a kernel bug, and not something specific to Proxmox.

 

[RedKage]

Can't comment on the nested virtualization issue as I haven't tried it.

Regarding docker, I have had zero problems running Linux VM's with CPU Type = host. Docker also runs fine on Linux VM's with CPU Type = host. Tested with multiple Debian 12 VM's running docker containers for 389-ds. The random crashes only occur when running Windows 11 VM's (haven't tried Windows 10 - though I suspect the same problem would occur).

There's obviously an issue with Zen 4 CPU's, KVM and virtualization of Windows 11 when using CPU type = host. I'd imagine it affects Windows Server as well, but have not yet tested it. It would seem to be a kernel bug, and not something specific to Proxmox.

Very interesting.
I did try a MacOS with CPU Host and also had some crashes. Especially during the install.
After switching to x86-64-v4, no crashes.

I never did any Linux VM so far, so your observations are very interesting.
I'm using LXC for all my Linux stuff.

I suspect also that proxmox is not to blame, but think the issue is on qemu's side though.
With a bad emulation for some CPU flags. Which maybe, the Linux kernel, doesn't use so it never gets emulated by qemu.

 

[jeenam]

Very interesting.
I did try a MacOS with CPU Host and also had some crashes. Especially during the install.
After switching to x86-64-v4, no crashes.

I never did any Linux VM so far, so your observations are very interesting.
I'm using LXC for all my Linux stuff.

I suspect also that proxmox is not to blame, but think the issue is on qemu's side though.
With a bad emulation for some CPU flags. Which maybe, the Linux kernel, doesn't use so it never gets emulated by qemu.

Also forgot to mention that I'm running a Windows 11 VM with GPU passthrough using an AMD RX 6600 XT. No issues, and it even passes EasyAnti-Cheat checks for Halo Infinite. Haven't tested any other games though.

 

[RedKage]

Also forgot to mention that I'm running a Windows 11 VM with GPU passthrough using an AMD RX 6600 XT. No issues, and it even passes EasyAnti-Cheat checks for Halo Infinite. Haven't tested any other games though.

This means you can GPU passthrough without CPU type Host?
Didn't know that. For some reason I thought you absolutely needed Host type CPU

 

[jeenam]

This means you can GPU passthrough without CPU type Host?

Yes

Didn't know that. For some reason I thought you absolutely needed Host type CPU

You can use any CPU Type you want with PCI/GPU Passthrough. Doesn't have to be 'host'.

 

[RedKage]

Yes


You can use any CPU Type you want with PCI/GPU Passthrough. Doesn't have to be 'host'.

Bro,
Thanks to you I just realized nested virtualization may be possible without Host type CPU too

Here's one guy who did it there https://forum.proxmox.com/threads/nested-virtualization-without-cpu-host.39102/

I am currently trying with these params for my VM

cpu: x86-64-v4,flags=+virt-ssbd;+amd-ssbd;+aes
args: -cpu max,kvm=on,vmware-cpuid-freq=on,+invtsc,+aes,+vmx

I hope this works and stays stable. I may finally be able to run WSL and docker in Win11...
Which means I will have no reason to even try to make CPU type Host work anymore.

At least for now Core Isolation is activated on this VM which wasn't the case before.
Time will tell. I will post back if I get crashes

 

[ccj47]

Hi I have found this thread due to the "exact" problem. When I stress the Waterfox by running browser benchmark or opening many tabs, the PVE host will reboot suddenly without graceful shutdown of any guest (and the host itself). It only happens recently so maybe some kernel updates introduced this problem to my specific configuration?

Linux 6.8.12-3-pve (2024-10-23T11:41Z)
pve-manager/8.2.7/3e0176e6bb2ade3b
Host CPU Ryzen 7700x
Guest CPU type: host
Guest OS Windows 11 Enterprise IoT LTSC 24H2
Waterfox G6.0.20

Switched Guest CPU type to x86-64-v4 and it seems to solve the issue. And the Win 11 seems to be more snappy. Shouldn't guest cpu host type be faster???

 

[RedKage]

I got some news.

So to recap, I need nested virtualization, and nothing was working so far.
1. Setting CPU type to x86-64, doesn't crashes, but nesting virt is unavailable
2. Adding args like args: -cpu max,kvm=on,vmware-cpuid-freq=on,+invtsc,+aes,+vmx makes virt works, but eventually crashes again (BTW only -cpu max is important, and only this alone creates crashes)
3. Setting CPU to host is the same, virt works but random crashes

Only one thing worked for me:
- Open /etc/modprobe.d/kvm-amd.conf
- Add vls=0
- Here is my full line options kvm-amd nested=1 vls=0

Now I can set CPU to host type, or put the above args with CPU x86-64 and I have virt available and no crashes.
Been testing this for 2 months now I think. Not a single crash, and I can enjoy WSL/Windows sandbox in my VM.

I am very curious as to why VLS needs to be disabled on this CPU.
I'd like to know where the issue lies, if that's the kernel, proxmox, qemu or the CPU hardware.

 

[RedKage]

Switched Guest CPU type to x86-64-v4 and it seems to solve the issue. And the Win 11 seems to be more snappy. Shouldn't guest cpu host type be faster???

That's because nested virtualization becomes unavailable when using x86-64-v4, and one consequence is that Win 11's "Core Isolation" will be disabled. This boosts a lot the performance, to the price of... less security according to M$.

You can have as fast as a system with host CPU + disabling core isolation.

 

[jeenam]

That's because nested virtualization becomes unavailable when using x86-64-v4, and one consequence is that Win 11's "Core Isolation" will be disabled. This boosts a lot the performance, to the price of... less security according to M$.

You can have as fast as a system with host CPU + disabling core isolation.

A kernel patch has been issued and is available in the Proxmox testing repo. The problem was a kernel bug.

https://forum.proxmox.com/threads/sudden-bulk-stop-of-all-vms.139500/post-718198

 

[ccj47]

I have updated PVE kernel to the 6.8.12-4 to fix the sudden bulk stop of all VMs on Zen4.

Then I switched the Windows 11 24H2 guest cpu type to host, ran Speedometer 3.0 benchmark with and without Core Isolation. With Core Isolation disabled, the guest OS ran the benchmark 3 times faster.