Windows Server 2025 CPU suddenly at 100%

Huh? Didn't you go with a workaround because it wouldn't fix the problem?

Code:
https://github.com/proxmox/qemu-server/commit/b69480d6110c005b9eb936c55c0438607d10975b

Well, even if this did fix 100% of the issues, enabling VBS would make the Start button on Windows unresponsive—it'd be completely useless. Shouldn't Proxmox just officially recommend disabling VBS at this point?

I don't think the issue that needs to be fixed is the one where it reaches 100%; rather, I think it's any problem that occurs when MBEC is used after VBS is enabled.

If anything, things are even worse now than they were on April 23, right?

Code:
https://forum.proxmox.com/threads/high-vm-exit-and-host-cpu-usage-on-idle-with-windows-server-2025.163564/post-849774

It wasn't exactly ideal, but it was a lot better than it is now.
 
Last edited:
A fix available now: https://pve.proxmox.com/wiki/Roadmap#9.2-known-issues (Point 3) "On Intel hosts, Windows 11/2022/2025 VMs with host CPU type and VBS enabled may intermittently freeze"
This isn't a conversation. (Though maybe you just don't want to talk to me.)
Enabling VBS doesn't cause CPU usage to reach 100%, but the Windows Start button stops responding, rendering it completely useless.

Is the Start menu a feature that isn't used in Windows?
 
@uzumo thanks for your comment.
What exactly does the error look like for you? Does the windows startmenü really stop responding completely after VBS activation, respond only very slowly, or does the menu pop up and then close right away?

And you're using an Intel CPU, right?

Please also post your VMconfig and the details about your CPU.

Code:
qm config <vmid>

Code:
lscpu

What Microsoft Windowsversion did you use?

Please also post the output of the following:

Code:
cat /proc/cmdline
pveversion --verbose

Shouldn't Proxmox just officially recommend disabling VBS at this point?
If you don't need the feature... well, then I recommend disabling it. Especially with Intel CPUs... It works much more better with AMD.
 
Last edited:
What exactly does the error look like for you? Does the windows startmenü really stop responding completely after VBS activation, respond only very slowly, or does the menu pop up and then close right away?
This sounds like an issue we faced here, please take a look:

They have used an Intel(R) Xeon(R) Bronze 3204 CPU and faced:
  • The Explorer shows up and I can click on apps in the taskbar,
    but when I try to open File Explorer or the Start Menu, it crashes and restarts.
  • The same behavior occurs when I left‑click on the desktop.
We fixed it by changing the cpu type to x86-64-v3.
 
@kfietz thanks for the info. I had a similar problem here on an HP ML350 Gen10 ( Intel Silver 4114 CPU @ 2.20GHz) when using x86-64-v4 for Windows Server 2025. Explorer didn't crash, but the Start menu would pop up and close immediately. This made it impossible to use the Windows Start menu. Switching to x86-64-v3 and now to “host” solved the problem here.
 
Thanks for your reply.
The command results are as follows. I added what seemed necessary since some information was probably missing, but feel free to skip this if you don't need it.

Code:
CPU : Intel Core Ultra 7 265K
MEM : Crucial CP2K48G56C46U5 x4
MB : Asrock Z890 Pro RS WiFi White
Windows Server 2025 Guest

Code:
root@pve1:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-7.0.14-8-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs iommu=pt split_lock_detect=off default_hugepagesz=1G hugepagesz=1G hugepages=92

Code:
root@pve1:~# cat /etc/default/grub | grep LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt split_lock_detect=off default_hugepagesz=1G hugepagesz=1G hugepages=92"

Code:
root@pve1:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
3607-D5AE is configured with: uefi (versions: BOOTX64.CSV, fbx64.efi, grub.cfg, grubx64.efi, mmx64.efi, shimx64.efi), grub (versions: 7.0.14-7-pve, 7.0.14-8-pve)

Code:
root@pve1:~# lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      20
  On-line CPU(s) list:       0-19
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Core(TM) Ultra 7 265K
    CPU family:              6
    Model:                   198
    Thread(s) per core:      1
    Core(s) per socket:      20
    Socket(s):               1
    Stepping:                2
    CPU(s) scaling MHz:      64%
    CPU max MHz:             5500.0000
    CPU min MHz:             800.0000
    BogoMIPS:                7756.80
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pg
                             e mca cmov pat pse36 clflush dts acpi mmx fxsr sse
                             sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm cons
                             tant_tsc art arch_perfmon pebs bts rep_good nopl xt
                             opology nonstop_tsc cpuid aperfmperf tsc_known_freq
                              pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm
                             2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
                             x2apic movbe popcnt tsc_deadline_timer aes xsave av
                             x f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault
                              epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow
                             flexpriority ept vpid ept_ad fsgsbase tsc_adjust bm
                             i1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx sma
                             p clflushopt clwb intel_pt sha_ni xsaveopt xsavec x
                             getbv1 xsaves split_lock_detect user_shstk avx_vnni
                              lam wbnoinvd dtherm ida arat pln pts hwp hwp_notif
                             y hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip
                             pku ospke waitpkg gfni vaes vpclmulqdq rdpid bus_lo
                             ck_detect movdiri movdir64b fsrm md_clear serialize
                              arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:
  Virtualization:            VT-x
Caches (sum of all):
  L1d:                       704 KiB (18 instances)
  L1i:                       1.1 MiB (18 instances)
  L2:                        36 MiB (11 instances)
  L3:                        30 MiB (1 instance)
NUMA:
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-19
Vulnerabilities:
  Gather data sampling:      Not affected
  Ghostwrite:                Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Old microcode:             Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via p
                             rctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user poi
                             nter sanitization
  Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditi
                             onal; PBRSB-eIBRS Not affected; BHI BHI_DIS_S
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace

Code:
root@pve1:~# pveversion --verbose
proxmox-ve: 9.2.0 (running kernel: 7.0.14-8-pve)
pve-manager: 9.2.5 (running version: 9.2.5/20242970da7fbcef)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-8-pve-signed: 7.0.14-8
proxmox-kernel-7.0: 7.0.14-8
proxmox-kernel-7.0.14-7-pve-signed: 7.0.14-7
ceph-fuse: 19.2.3-pve2
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.19
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.7
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
openvswitch-switch: 3.5.0-1+b1
proxmox-backup-client: 4.2.3-1
proxmox-backup-file-restore: 4.2.3-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.6
pve-cluster: 9.1.6
pve-container: 6.1.12
pve-docs: 9.2.3
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.5
pve-firmware: 3.18-5
pve-ha-manager: 5.2.5
pve-i18n: 3.9.0
pve-qemu-kvm: 11.0.3-1
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.1
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.3-pve1

This occurs in any of the following situations. It either doesn't work or suddenly starts working?I have no idea what's going on. As long as VBS is disabled, this never happens.

It’s ridiculous that we have to edit this kind of thing every time there’s a minor update. And, in reality, we aren’t provided with the information needed to make the edits.

Do you understand that it’s putting the cart before the horse to enable MBEC for performance gains and then specify x86-64-v3?which results in reduced functionality compared to the host (for Haswell-generation CPUs)?

“As long as it works, that's good enough”?when it comes to toys, that's probably acceptable.

If you haven't identified what's causing the problem and what would make it work, then it just happened to work by chance?and that isn't even a solution.

"Status: RESOLVED FIXED" Ha ha ha

Code:
root@pve1:~# cat /etc/pve/virtual-guest/cpu-models.conf
cpu-model: Nested
        flags +invtsc;-cet-ibt;-cet-ss;-hypervisor;+pdpe1gb;+vmx
        hv-vendor-id test
        phys-bits 39
        reported-model max

Code:
root@pve1:~# cat /etc/pve/virtual-guest/cpu-models.conf
cpu-model: Nested
        flags +invtsc;-cet-ibt;-cet-ss;-hypervisor;+pdpe1gb;+vmx
        hv-vendor-id test
        phys-bits 39
        reported-model host

Code:
root@pve1:~# cat /etc/pve/virtual-guest/cpu-models.conf
cpu-model: Nested
        flags +invtsc;-cet-ibt;-cet-ss;+pdpe1gb;+vmx
        hv-vendor-id test
        phys-bits 39
        reported-model host

Code:
root@pve1:~# sudo qm showcmd 1171 --pretty true | grep cpu
  -smp '4,sockets=1,cores=4,maxcpus=4' \
  -cpu 'host,level=30,-cet-ibt,-cet-ss,hv_evmcs,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vendor_id=test,hv_vpindex,-hypervisor,+invtsc,+kvm_pv_eoi,+kvm_pv_unhalt,+pdpe1gb,+vmx,phys-bits=39' \

Code:
root@pve1:~# sudo qm showcmd 1171 --pretty true | grep cpu
  -smp '4,sockets=1,cores=4,maxcpus=4' \
  -cpu 'host,level=30,-cet-ibt,-cet-ss,hv_evmcs,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vendor_id=test,hv_vpindex,+invtsc,+kvm_pv_eoi,+kvm_pv_unhalt,+pdpe1gb,+vmx,phys-bits=39' \

Code:
root@pve1:/etc/pve/qemu-server# cat 1171.conf
cores: 4
cpu: custom-Nested,level=30
machine: pc-q35-11.0+pve2

As of April, MBEC was working fine even in the following scenarios, but since the patch was applied, it's been a disaster. There's no way to hide the fact that it's a virtual machine.

Code:
args: -cpu max,migratable=off,-cet-ibt,-cet-ss,hv_passthrough,-hypervisor,hv-vendor-id=0123456789AB,level=30,+vmx,invtsc=on,guest-phys-bits=39 -global intel-iommu.aw-bits=39

> If you don't need the feature... well, then I recommend disabling it. Especially with Intel CPUs... It works much more better with AMD.

That really made me laugh. I can’t believe Proxmox would say something like that. It's as if they're saying, “I give up?I've had enough.”

At the end of the day, your comment just proves that this is merely a stopgap measure, not a perfect solution. It works fine on AMD, but they just made it work on Intel for now, right?
Given that, it’s really ridiculous to claim there’s a solution.

Since no measures have been taken for the future, are they really trying to claim this is a solution? But doesn’t your statement actually show that the solution is flawed?
 
Last edited:
A fix available now: https://pve.proxmox.com/wiki/Roadmap#9.2-known-issues (Point 3) "On Intel hosts, Windows 11/2022/2025 VMs with host CPU type and VBS enabled may intermittently freeze"
Not sure if you prefer this here or in another thread but I installed updates today on all nodes. On a VM using machine type 11.0 and CPU x86-64-v3, I changed it to 11.0+pve2. When I restarted it via PVE GUI, it fails to start on or migrate to its original host:

Code:
/dev/rbd20
/dev/rbd21
/dev/rbd22
/dev/rbd23
netdev net0: using 'host_mtu=1500' for migration compat
QEMU: kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
stopping swtpm instance (pid 18298) due to QEMU startup error

TASK ERROR: start failed: QEMU exited with code 1


It does run on (automatically migrated to) another node which should be identical hardware (Xeon Silver 4510, Linux 7.0.14-8-pve). Other VMs were version 10.1 so I left them unchanged.

Any ideas why this one is complaining?
 
One would think but I upgraded them about 15 minutes apart.

The one that works has a few extra older kernel versions as I hadn't run autoremove yet. I use the PVE GUI to upgrade. "uname -a" is the same on both. Which might imply some hardware difference but we ordered them like a week apart with the same config. BIOS, somehow?

pve-manager: 9.2.4 (running version: 9.2.4/5e5ae681198514d4)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-8-pve-signed: 7.0.14-8
proxmox-kernel-7.0: 7.0.14-8
proxmox-kernel-7.0.14-4-pve-signed: 7.0.14-4
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
proxmox-kernel-6.17.13-16-pve-signed: 6.17.13-16
proxmox-kernel-6.8.12-20-pve-signed: 6.8.12-20
proxmox-kernel-6.8: 6.8.12-20
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph: 20.2.2-pve1
ceph-fuse: 20.2.2-pve1
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.16
libpve-guest-common-perl: 6.0.4
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.6
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.3-1
proxmox-backup-file-restore: 4.2.3-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.6
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.3
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.5
pve-firmware: 3.18-5
pve-ha-manager: 5.2.4
pve-i18n: 3.9.0
pve-qemu-kvm: 11.0.2-1
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.1
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.3-pve1
 
@SteveITS mhh so the other nodes are running the same kernel.
The existing older Kernels do not really matter.

Maybe some different BIOS Settings? Intel Nested Virtualization maybe off? But v3 shouldnt even use that, if I understood that correctly. Did you add any flags manually?

If it is a BIOS setting, you can maybe check the available CPU flags?

cat /proc/cpuinfo

Do they differ between the nodes?
Is nested enabled on both kernels?

cat /sys/module/kvm_intel/parameters/nested
 
Indeed, nested is not enabled on the "problem" node. CPU flags are the same.

Leaving two questions I think:
1) should PVE/QEMU fail this VM startup if nested is disabled, or just ignore that and proceed? (it's saying to use a flag that it can't, but it doesn't need to...?)

2) is the correct move to turn on nested on this one, or turn off nested on the "good" node? Other than Windows' built-in security, we have no intention of nesting. I am wondering if that would leave us stuck on machine type 10.1 or lower.

Thanks.
 
As it even fails to start fresh on this "problem" node, then maybe the check for hv-evmcs is forced, even tho it is not used by the cpu type later on?
Maybe it is added by defining the os type as windows?

Is it listed under cpu features in -cpu with command:

qm showcmd <vmid> --pretty

Maybe you should consider enabling nested virtualization.
https://pve.proxmox.com/wiki/Nested_Virtualization#Enable_Nested_Hardware-assisted_Virtualization
Weird that it is disabled in the first place. Normally it is enabled by default. Anything in /proc/cmdline or /etc/modprobe.d/ that disables it?
Not aware of a BIOS Setting where you can disable nested specifically, but definitely worth a check.