High VM-EXIT and Host CPU usage on idle with Windows Server 2025

[SchorschSK]

@_gabriel exactly as @RoCE-geek said.


Hardware: 2x Xeon 6230R and both VMs had this config.

agent: 1
bios: ovmf
boot: order=scsi0;ide2;ide0;net0
cores: 52
cpu: Cascadelake-Server-v5,flags=+md-clear;+pcid;+spec-ctrl;+pdpe1gb;+hv-tlbflush;+hv-evmcs
efidisk0: linstor_nvme_1:pm-33087c22_114,efitype=4m,ms-cert=2023,pre-enrolled-keys=1,size=3080K
hotplug: 0
ide0: ISO-1:iso/virtio-win-0.1.285.iso,media=cdrom,size=771138K
ide2: ISO-1:iso/Win11_23H2_English_x64.iso,media=cdrom,size=6548134K
machine: pc-q35-10.1
memory: 64000
meta: creation-qemu=10.1.2,ctime=1764535929
name: win11-23h2
net0: virtio=BC:24:11:E4:D1:51,bridge=vmbr0,firewall=1
numa: 1
ostype: win11
scsi0: linstor_nvme_1:pm-a0c6d8aa_114,discard=on,iothread=1,size=159383560K,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
sockets: 2
tpmstate0: linstor_nvme_1:pm-29d19dc3_114,size=4M,version=v2.0
vga: virtio

Win23H2 is only marginally faster (margin of error) on all cores but the single core speedup is 18% which kinda reflects the max boost frequency during the single core test run.

Xeon 6230R was able to boost up to 3.75GHz in Win11 23H2 vs 3.1GHz turbo boost on Win11 25H2.

 

[MarkusKo]

Just for reference, updating PVE to 9.1 / Kernel 6.17 did reduce my CPU usage for VM's, including Windows Server 2025
Node CPU: AMD Epyc 9355P, VM Processor Type: x86-64-v4, virtio-win-0.1.271

Node1 Windows Server 2025 VM CPU usage:



Node2 Windows Server 2025 VM CPU usage:

 

[RoCE-geek]

Just for reference, updating PVE to 9.1 / Kernel 6.17 did reduce my CPU usage for VM's, including Windows Server 2025
Node CPU: AMD Epyc 9355P, VM Processor Type: x86-64-v4, virtio-win-0.1.271

Node1 Windows Server 2025 VM CPU usage:
View attachment 93485


Node2 Windows Server 2025 VM CPU usage:
View attachment 93487

Thanks, will check the impact. What's your VM version? Was the PVE upgrade itself only one change made?

 

[MarkusKo]

Version is 10.0+pve1. I did only update PVE, did not modify the VM's.

 

[guruevi]

Here is a reference to the documentation: https://www.qemu.org/docs/master/system/i386/hyperv.html

The fallback that people noticed to other timers is desired. Yes, Windows 2025/11 instituted more timers because Windows software is broken and may not function well in VM, so this is a kernel change in Windows causing more interrupts. You can play with the above switches to see if you can get better emulation of HyperV but even there is higher CPU usage.

Do qm showcmd, see if hv_frequencies is in there and post the output here. Add hv_frequencies to args if necessary.

 

[RoCE-geek]

Here is a reference to the documentation: https://www.qemu.org/docs/master/system/i386/hyperv.html

The fallback that people noticed to other timers is desired. Yes, Windows 2025/11 instituted more timers because Windows software is broken and may not function well in VM, so this is a kernel change in Windows causing more interrupts. You can play with the above switches to see if you can get better emulation of HyperV but even there is higher CPU usage.

Do qm showcmd, see if hv_frequencies is in there and post the output here. Add hv_frequencies to args if necessary.

I've been experimenting with many of these flags - no positive change with hv_frequencies either. There's no (clear) solution via HVE flags.

 

[nodoame]

No huge difference for me after updating from kernel 6.16.8 to 6.17.8 (plain Debian) unfortunately. Still the same 2-3x higher idle load and no turbo boost...

 

[guruevi]

I'm kind of wondering what people are expecting here after reading some more into the above details - you have 8-16 CPUs, according to the documentation, it is expected that HyperV guests will use less than 1% of CPU per core when 'idle'. So the total host CPU usage should be under 8-16% of 1 CPU at 'idle' (tasks may not get distributed to multiple cores at idle). I get ~5% for a 4-core Windows 11 guest and that seems to scale up as I add more cores. Note that setting the guest type to Linux in Proxmox brings that idle usage to ~15-20%, you have to set the guest type to Windows 11/2025 it drops to ~4-5% and this is on a relatively busy network with AD etc, so lots of things are happening 'constantly' to Windows machines. Look at the logs, you can disable a lot of services to get "more idle", but Windows is quite chatty compared to even 'fat' Linux distros. I get 0.23% on idle containers and ~1% on idle Linux (RHEL, Ubuntu).

 

[RoCE-geek]

I'm kind of wondering what people are expecting here after reading some more into the above details - you have 8-16 CPUs, according to the documentation, it is expected that HyperV guests will use less than 1% of CPU per core when 'idle'. So the total host CPU usage should be under 8-16% of 1 CPU at 'idle' (tasks may not get distributed to multiple cores at idle). I get ~5% for a 4-core Windows 11 guest and that seems to scale up as I add more cores. Note that setting the guest type to Linux in Proxmox brings that idle usage to ~15-20%, you have to set the guest type to Windows 11/2025 it drops to ~4-5% and this is on a relatively busy network with AD etc, so lots of things are happening 'constantly' to Windows machines. Look at the logs, you can disable a lot of services to get "more idle", but Windows is quite chatty compared to even 'fat' Linux distros. I get 0.23% on idle containers and ~1% on idle Linux (RHEL, Ubuntu).

We know quite well what is happening and what has changed in 24H2 core (aka WS2025/Win11-24H2). Just check the recent evidence-based posts if really interested in the root cause. There's a stable WS2022/Win11-23H2 idle baseline, with sudden change in 24H2 core, which is caused by sudden abuse of specific synthetic timers (and/or other substitution VM calls if disabled). This is all we're talking about - and how to mitigate it, otherwise this change will seriously impact any existing dense VM environment. This is not about theorizing Windows vs Linux or about disabling Windows services, etc.

 

[guruevi]

@RoCE-geek: I'm not seeing the above as a performance issue on my hardware (rather modern Xeon Gold) in production environments with current Win11-25H2 with mostly default recommended settings.

Even looking at the other posts, I see a 'normal' looking Windows 11/2025 system. I can see where if you set the settings to Linux which disables the Hyper-V emulation, then, the idle jumps to 20%, which would be significant. I can understand AMD virtualization may be worse than Intel. But I'm looking at the documentation from Microsoft where this is discussed and Microsoft says this is 'normal' even on Hyper-V.

But this is VMware cluster for an idle Windows Server host (average it says is ~5%):


Hyper-V conveniently 'hides' VMs from its host statistics, so without some Prometheus-fu I'm not going to even try to understand, but this is a bare metal Windows (on PowerEdge), "idle" CPU usage bounces to ~10% max according to iDRAC, with an average of, again ~4%.


If you're seeing 4-5% CPU usage or a little higher on older models, I think we're chasing ghosts trying to get it to 1-2% we're "used to" in Linux-land because even the bare metal, Windows is just yes, triggering timers, which costs CPU time and has to be patched through. And KVM is not alone in finding out this, various other audio equipment and real-time software and games is "noticing" significant performance issues with interrupts for current iterations of Windows 11, it's not something that "we" could fix without Microsoft.

 

[RoCE-geek]

@RoCE-geek: I'm not seeing the above as a performance issue on my hardware (rather modern Xeon Gold) in production environments with current Win11-25H2 with mostly default recommended settings.

Even looking at the other posts, I see a 'normal' looking Windows 11/2025 system. I can see where if you set the settings to Linux which disables the Hyper-V emulation, then, the idle jumps to 20%, which would be significant. I can understand AMD virtualization may be worse than Intel. But I'm looking at the documentation from Microsoft where this is discussed and Microsoft says this is 'normal' even on Hyper-V.

But this is VMware cluster for an idle Windows Server host (average it says is ~5%):
View attachment 93520

Hyper-V conveniently 'hides' VMs from its host statistics, so without some Prometheus-fu I'm not going to even try to understand, but this is a bare metal Windows (on PowerEdge), "idle" CPU usage bounces to ~10% max according to iDRAC, with an average of, again ~4%.
View attachment 93517

If you're seeing 4-5% CPU usage or a little higher on older models, I think we're chasing ghosts trying to get it to 1-2% we're "used to" in Linux-land because even the bare metal, Windows is just yes, triggering timers, which costs CPU time and has to be patched through. And KVM is not alone in finding out this, various other audio equipment and real-time software and games is "noticing" significant performance issues with interrupts for current iterations of Windows 11, it's not something that "we" could fix without Microsoft.

And here is the root of "misunderstanding". Most of the contributors here are looking for the 24H2 side-effects mitigation, simply because a 3x increase of idle load is hardly acceptable for us, and we do not want to resign ourselves to the fact that this is the "new standard" (if you do, good for you). We've analyzed the root cause and we're definitely not alone - other platforms like FreeBSD/Bhyve seem to be affected in even higher degree.

So why are we so interested? At least for these main reasons: 1) with WS2022 baseline, there is no real/impactful added idle CPU load, 2) other KVM-based system already mitigated this problem (Nutanix here), 3) we're very well aware of the impact of any massive WS2022->WS2025 upgrades in our infrastructure, hence wanna be prepared in advance, and 4) if something technically deactivates TurboBoost, this is definitely bug, not acceptable feature.

That's all - It won't help any of us here to pat each other on the back and say that it's actually okay and that we should just leave it alone.

 

[SchorschSK]

mentioning the big boyz: VmWare ESX 8 is also fine...

can somebody test hyper-v please? windows is not my piece of cake...