High VM-EXIT and Host CPU usage on idle with Windows Server 2025

@Jostein Fossheim
steve72 said:
If you instead of cpu model host use x86-64-v3 or something similar performance will be much better.
Click to expand...
because nested virtualization isn't available for VM.
This is the way ^^

steve72 said:
You can also disable nested virtualization and continue to use host as cpu model and performance is good again.
Click to expand...
This is the other way to keep host as vCPU type, and you don't want to use.
 
I've revisited VBS / credencial guard on Proxmox guests on our AMD cluster many times over the years. Have no seen much improvement on our cluster...

When set to "host" CPU type, Windows 11/2025 will automatically attempt to enable VBS with nested extensions available, which causes high idle CPU utilization and laggy interactive performance in the VM. Our windows domain has VBS/cred-guard enabled by group policy as well.

I stumbled on this yesterday while researching this...

https://williamlam.com/2023/07/vsph...eneration-2-nested-vm-running-on-amd-cpu.html

I suspect there's a similar problem going on with Proxmox, on certain hardware platforms?
 
Same here, Proxmox 8.3.5, Windows Server 2025 Datacenter with all available Windows updates, latest virtio drivers (266).

I freshly installed Win2025 on a already existent VM (Win2022 was installed before, I wiped the virtual disk), so I can easily see that there is a huge difference between 2022 and 2025.
The peak was the installation phase and reboots, in the afternoon the server was on idle.

1743093804654.png
 
_gabriel said:
This is the cost of nested virtualization used by VBS.
Easier is switch vCPU type to x86-64-v2-aes or v3 if newer physical CPU.
What is the physical host CPU model ?
Click to expand...
Host CPU on this test machine is a i9-13900. Will check on the VBS topic.
 
_gabriel said:
This is the cost of nested virtualization used by VBS.
Easier is switch vCPU type to x86-64-v2-aes or v3 if newer physical CPU.
What is the physical host CPU model ?
Click to expand...
Unfortunately, this isn't a valid option for modern enterprise environments. VBS and associated technologies are effectively required by contractual obligation for many businesses these days. There are security controls and configuration baselines that require it.

My understanding is that modern CPUs have nested virtualization/paging capabilities that should allow this to work with very little performance compromise. It seems to me like there's just a implementation problem/bug in KVM/QEMU preventing it from working as intended.
 
  • Like
Reactions: orange, Whatever and Taomyn
I went ahead and tried the following:
- Disabled nested virtualization using kvm-intel.conf, rebooted the server
- Set the virtual machine to use x86-64-v4 and x86-64-v2-AES

Unfortunately, that did not change the performance characteristics related to idle cpu interrupts on the Windows Server 2025 virtual machine.

1743962438194.png

msinfo32
1743962761525.png
 
@orange Did you find a fix yet?

I am having the same issue with a Windows Server 2022 VM upgraded to Windows Server 2025 on an Intel Core i7-12700 host.
It is idling at about 2-3% CPU usage vs < 1% on WS2022 with 8 cores configured as host.

in powertop, on the host, Pkg(HW) never enters any C-state when WS2025 is running. With WS2022 running, it spends about 10-20% of it's time in C2 (idle).

VBS is disabled and the WS2025 VM itself is quite snappy via RDP.
 
_gabriel said:
Have you tried as vCpu type x86-64-v2-aes ?
Click to expand...

Yes, I forgot to mention it, as it's been referenced earlier in the thread. I tried x86-64-v2-AES and x86-64-v3. Results were the same.

I did not however try to thinker with spec-ctl / pcid flags, as it is unclear to me if I should select minus (-) or plus (+) to have an effect.
 
Just encountered the same issue after upgrading a Win 10 Pro VM to Win 11 Pro. VBS off and followed all advices from this thread. Still having idle percentage 2 to 3 times higher than before the upgrade. Inside the VM, there is virtually no CPU usage, except I am seeing more time spent with interrupts that it used to. Perhaps a virtio driver issue? Did you find any solution in the meantime?
 
I haven't found a solution yet on my side. I spent quite some time on the issue and I was able to completely disable spectre mitigation on the host (pve), in the guest VM, disable power management, make sure vbs is disabled, etc but it did not make a significant improvement. It looks like the newer kernels handles idle cores differently.

Those are my host interrupts on Windows 2022. Note that only the first vCore generates events while the VM is idle.
Usage Events/s Description
6.1 ms/s 33.8 [PID 2616656] /usr/bin/kvm -id 116 -name vm-windows2022
7.0 ms/s 0.7 [PID 2616657] /usr/bin/kvm -id 116 -name vm-windows2022
5.9 ms/s 1.1 [PID 2616661] /usr/bin/kvm -id 116 -name vm-windows2022
4.2 ms/s 1.6 [PID 2616659] /usr/bin/kvm -id 116 -name vm-windows2022
Click to expand...

Those are interrupts on Windows 2025. It is the same base VM which has been upgraded from Windows 2022 and is idle / up to date.

Usage Events/s Description
16.8 ms/s 221.1 [PID 2622138] /usr/bin/kvm -id 110 -name vm-windows2025
13.8 ms/s 201.0 [PID 2622140] /usr/bin/kvm -id 110 -name vm-windows2025
11.8 ms/s 180.3 [PID 2622139] /usr/bin/kvm -id 110 -name vm-windows2025
11.2 ms/s 130.5 [PID 2622137] /usr/bin/kvm -id 110 -name vm-windows2025
Click to expand...

It seems the newer kernel generates interrupts on each cores whether they're idle or not.
I tried increasing the vCore count to 8 and they also each get about 100-200 events/s.

In the end, it increases the power consumption by 2W on my host, so I stopped chasing the issue.
 
I can reproduce this on a plain Debian testing machine using fresh Windows 11 24H2 / 25H2. 23H2 / Windows 10 22H2 are unaffected.
Unfortunately, I can't just ignore this as not only does power consumption increase but those interrupts also prevent CPU cores from entering C3 state which in turn deactivates Intel Turbo Boost.

Code: 
# 23H2 (cpuset=uneven cores)
$ perf kvm --host stat live
                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                             EOI_INDUCED        494    48.96%     0.03%      1.43us      6.00us      4.04us ( +-   0.72% )
                                     HLT        492    48.76%    98.96%   4530.68us  15769.42us  15435.74us ( +-   0.32% )
                           EPT_MISCONFIG          9     0.89%     1.02%     13.07us  77606.03us   8658.88us ( +-  99.53% )
                      EXTERNAL_INTERRUPT          5     0.50%     0.00%     14.94us     20.75us     17.80us ( +-   5.33% )
                                  VMCALL          5     0.50%     0.00%      4.66us     14.60us      8.37us ( +-  20.82% )
                              APIC_WRITE          4     0.40%     0.00%      5.20us      7.13us      6.41us ( +-   6.52% )

$ turbostat --quiet --interval 1
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     NMI     SMI     POLL%   C1%     C2%     C3%     CPU%c1  CPU%c6  CPU%c7  CoreTmp CoreThr PkgTmp  Totl%C0 Any%C0  GFX%C0  CPUGFX% Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc8 Pk%pc10 CPU%LPI SYS%LPI PkgWatt CorWatt GFXWatt RAMWatt PKG_%   RAM_%   UncMHz  SysWatt
-       -       13      1.64    813     2112    0.50    2892    0       0       0.00    1.05    6.96    90.47   3.33    38.03   55.69   39      0       40      37.05   22.30   0.00    0.00    57.25   0.00    0.00    0.00    0.00    0.00    0.00    2.36    1.10    0.00    0.00    0.00    0.00    800     0.00
0       0       11      1.43    800     2112    0.29    219     0       0       0.00    1.86    14.14   82.81   4.24    20.95   71.07   38      0       40      37.04   22.29   0.00    0.00    57.24   0.00    0.00    0.00    0.00    0.00    0.00    2.36    1.10    0.00    0.00    0.00    0.00    800
0       1       15      1.90    800     2112    0.34    200     0       0       0.00    1.34    10.45   86.56   4.23
4       2       13      1.68    801     2112    0.27    182     0       0       0.00    0.73    14.36   83.42   4.07    15.54   77.12   39      0
4       3       11      1.40    801     2112    0.27    140     0       0       0.00    0.35    7.95    90.50   4.07
8       4       17      1.97    859     2112    0.57    186     0       0       0.00    1.74    12.11   84.36   6.21    11.48   78.55   37      0
8       5       14      1.62    860     2112    0.64    128     0       0       0.00    0.33    4.39    93.82   6.21
12      6       11      1.41    801     2112    0.30    132     0       0       0.00    0.12    7.93    90.69   1.74    9.54    85.70   35      0
12      7       11      1.41    801     2112    0.29    122     0       0       0.00    0.25    4.95    93.55   1.74
16      8       6       0.75    806     2112    0.27    72      0       0       0.00    0.90    4.16    94.28   1.71    4.73    91.84   36      0
16      9       8       1.03    802     2112    0.31    98      0       0       0.00    0.26    2.19    96.62   1.71
20      10      5       0.60    803     2112    0.28    55      0       0       0.00    1.22    2.81    95.45   2.00    3.41    93.05   35      0
20      11      8       1.02    802     2112    0.32    96      0       0       0.00    0.37    2.09    96.61   2.00
24      12      15      1.72    867     2112    0.92    135     0       0       0.00    0.48    8.45    89.48   3.39    8.25    85.30   37      0
24      13      11      1.27    857     2112    0.28    112     0       0       0.00    0.17    4.69    94.00   3.39
28      14      12      1.45    805     2112    0.48    115     0       0       0.00    0.50    9.68    88.52   2.00    9.49    85.76   37      0
28      15      10      1.21    803     2112    0.30    115     0       0       0.00    0.18    4.29    94.47   2.00


# 24H2 (cpuset=uneven cores)
$ perf kvm --host stat live
                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                                     HLT       1105    49.46%    97.89%     20.43us  80025.30us   1827.12us ( +-   5.43% )
                             EOI_INDUCED       1099    49.19%     1.26%      0.64us  24351.12us     23.61us ( +-  93.83% )
                              APIC_WRITE         12     0.54%     0.00%      2.39us      4.90us      3.88us ( +-   6.18% )
                                  VMCALL          9     0.40%     0.00%      7.35us     16.19us     11.09us ( +-  10.53% )
                      EXTERNAL_INTERRUPT          7     0.31%     0.00%      4.24us     12.82us      6.83us ( +-  15.77% )
                           EPT_MISCONFIG          2     0.09%     0.84%     63.00us  17298.67us   8680.83us ( +-  99.27% )

$ turbostat --quiet --interval 1
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     NMI     SMI     POLL%   C1%     C2%     C3%     CPU%c1  CPU%c6  CPU%c7  CoreTmp CoreThr PkgTmp  Totl%C0 Any%C0  GFX%C0  CPUGFX% Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc8 Pk%pc10 CPU%LPI SYS%LPI PkgWatt CorWatt GFXWatt RAMWatt PKG_%   RAM_%   UncMHz  SysWatt
-       -       39      3.08    1277    2112    1.30    9082    0       0       0.01    2.27    44.87   49.88   78.11   29.66   0.00    39      0       43      64.32   35.65   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    5.53    4.04    0.00    0.00    0.00    0.00    800     0.00
0       0       17      1.73    958     2112    1.15    344     0       0       0.03    2.13    31.74   64.50   96.13   0.00    0.00    39      0       43      64.32   35.65   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    5.53    4.04    0.00    0.00    0.00    0.00    800
0       1       23      2.84    811     2112    1.01    871     0       0       0.00    2.44    94.89   0.00    96.12
4       2       3       0.38    815     2112    0.70    101     0       0       0.00    2.64    3.22    93.78   97.82   0.00    0.00    39      0
4       3       19      2.36    803     2112    1.10    855     0       0       0.00    1.43    96.37   0.00    97.83
8       4       11      1.06    1045    2112    0.87    158     0       0       0.03    2.15    10.54   86.29   96.81   0.00    0.00    37      0
8       5       23      2.72    829     2112    1.40    851     0       0       0.00    1.38    96.07   0.00    96.81
12      6       5       0.52    962     2112    0.76    88      0       0       0.00    2.82    5.10    91.58   97.74   0.00    0.00    36      0
12      7       19      2.30    806     2112    1.10    851     0       0       0.00    1.74    96.13   0.00    97.74
16      8       8       0.76    983     2112    0.69    142     0       0       0.00    3.16    6.56    89.55   97.32   0.00    0.00    38      0
16      9       20      2.52    804     2112    1.04    850     0       0       0.03    1.86    95.79   0.00    97.32
20      10      4       0.50    849     2112    0.91    203     0       0       0.00    6.70    4.13    88.71   97.58   0.00    0.00    38      0
20      11      20      2.48    806     2112    1.08    850     0       0       0.00    1.87    95.81   0.00    97.58
24      12      31      1.96    1580    2112    1.46    228     0       0       0.00    7.63    13.06   77.40   96.18   0.00    0.00    37      0
24      13      21      2.50    833     2112    1.01    849     0       0       0.03    1.57    96.10   0.00    96.18
28      14      9       1.03    915     2112    0.87    121     0       0       0.01    2.90    14.19   81.93   97.14   0.00    0.00    39      0
28      15      20      2.44    805     2112    1.05    851     0       0       0.03    1.47    96.26   0.00    97.14

Here I'm assigning the VM to all uneven cores which for some reason decreases C3 time from about 90% to exactly 0, despite CPU load only increasing by 1% or so per core on 24H2. Due to how Turbo Boost works this basically reduces single thread performance for the whole host with every additional vCPU on a 24H2+ VM.

I've tried a bunch of things like different CPU models, flags, enlightments, GlobalTimerResolutionRequests, bcdedit flags, disabling nested virtualization etc. but without much success.
The only way I found C3 to work is by enabling useplatformticks with HPET enabled. Unfortunately this results in even more interrupts and even larger CPU consumption in idle. This makes me believe something related to timer resolution changed between 23H2 and 24H2 but I'm out of ideas what else to try...
 
Last edited:
  • Like
Reactions: copec and SleepyCircuit
I haven’t virtualized Windows for years, but am doing a job for someone that requires a Windows 2025 Server VM, and have encountered the same thing. I have tried every combination of settings suggested in this thread. I see an 8-core windows 2025 server image take about 4-5% CPU as reported by PVE, or roughly 50% of a single core when the whole qemu process is added up. An 8-core vanilla Debian 13 image takes up 0.1% when idle on the same servers.
 
benyamin said:
You can try this: https://forum.proxmox.com/threads/win-server-2k25-qemu-disk-ultra-slow.176153/post-818148
Click to expand...
I set it up Windows 2025 with Q35, x86-64-v3, UEFI, TPM v2. VirtIO disks+NIC. The actual performance inside the VM seems great. It's just the symptoms of this exact thread title: When the VM is internally idle, the QEMU process tree takes up about 4% CPU usage per core, perf trace shows a lot more VM-EXIT than the other VM counterparts - there is an old Windows Server 2012 VM on another host I'm looking at, and it takes virtually no CPU idle, same with all the Linux VMs. System Information shows that Virtualization-based security is Not enabled. I figured this was just the state that QEMU/KVM was at and that there will be eventual fixes or understanding of how to set it. Going to watch the video in that thread you linked.
 
For the issue I called out (Memory Integrity part of VBS), if you do not have VBS, or nesting, exposed a VM, the issue isn't that
 
  • Like
Reactions: benyamin
I'm just sharing my perf stats, regardless it'd be useful or not.
For me it's hard to determine if I'm "affected" (by the increased idle load or other perf problems).
I've no other WS2025 VM stats to compare, as well as I cannot simply test it on Intel.

Testing setup: Proxmox 8.4.0 (running kernel: 6.8.12-15-pve), qemu-server 8.4.3 and four WS2025 VMs with CPU setting "EPYC-Milan-v2". I also use "high performance" profile setting in BIOS, as well as scaling_governor=performance. And with these VM and system settings, idle host-side CPU load is under 2% for 4x WS2025 VMs, running on 16C EPYC3 processor (16C/32vCPUs, 3.2GHz base clock). Guest power management is "High performance".

VM details: 4 VMs running, all WS2025 (6+6+8+8 = 28 vCPUs occupied, out of 32), all in (almost) idle state. Technically 3 of these VMs are "real" VMs, but at that moment there was no impactful load. For this reason I've been also waiting some time to catch the lowest possible values. In other (idle) cases there was more often 60/40 MSR/HLT ratio (with MSR 16.000-18.000).

I've experimentally concluded (by switching VMs ON/OFF), that any active (idle, but running) WS2025 vCPU consumes around 550-600 MSR events here, so what you see is 28 vCPUs running in total, within four WS2025 VMs (running, but all almost idle):

Code: 
         VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

             msr      15323    68.74%    32.99%      1.84us  47349.95us    413.92us ( +-   5.23% )
             hlt       6648    29.82%    66.41%      1.47us  49426.35us   1920.90us ( +-   0.94% )
       interrupt        160     0.72%     0.07%      1.63us  13357.33us     88.83us ( +-  93.94% )
       write_cr8         61     0.27%     0.04%      0.97us   7741.68us    128.75us ( +-  98.55% )
           vintr         33     0.15%     0.02%      1.29us   4028.20us    123.92us ( +-  98.45% )
       hypercall         27     0.12%     0.00%      2.56us     15.92us      5.12us ( +-  10.49% )
              io         20     0.09%     0.28%      6.13us  23578.13us   2675.70us ( +-  58.43% )
             npf         20     0.09%     0.19%      4.59us  12490.88us   1788.86us ( +-  54.34% )
 
You must log in or register to reply here.
Top Bottom