High VM-EXIT and Host CPU usage on idle with Windows Server 2025

[_gabriel]

@Jostein Fossheim

If you instead of cpu model host use x86-64-v3 or something similar performance will be much better.

because nested virtualization isn't available for VM.
This is the way ^^

You can also disable nested virtualization and continue to use host as cpu model and performance is good again.

This is the other way to keep host as vCPU type, and you don't want to use.

 

[Jostein Fossheim]

I can also add that no matter how many cores i give the VM (between 2 or 40) it uses the same idle load between 2-3%.

 

[AllanM]

I've revisited VBS / credencial guard on Proxmox guests on our AMD cluster many times over the years. Have no seen much improvement on our cluster...

When set to "host" CPU type, Windows 11/2025 will automatically attempt to enable VBS with nested extensions available, which causes high idle CPU utilization and laggy interactive performance in the VM. Our windows domain has VBS/cred-guard enabled by group policy as well.

I stumbled on this yesterday while researching this...

https://williamlam.com/2023/07/vsph...eneration-2-nested-vm-running-on-amd-cpu.html

I suspect there's a similar problem going on with Proxmox, on certain hardware platforms?

 

[ksb]

Same here, Proxmox 8.3.5, Windows Server 2025 Datacenter with all available Windows updates, latest virtio drivers (266).

I freshly installed Win2025 on a already existent VM (Win2022 was installed before, I wiped the virtual disk), so I can easily see that there is a huge difference between 2022 and 2025.
The peak was the installation phase and reboots, in the afternoon the server was on idle.

 

[_gabriel]

huge difference between 2022 and 2025

This is the cost of nested virtualization used by VBS.
Easier is switch vCPU type to x86-64-v2-aes or v3 if newer physical CPU.
What is the physical host CPU model ?

 

[ksb]

This is the cost of nested virtualization used by VBS.
Easier is switch vCPU type to x86-64-v2-aes or v3 if newer physical CPU.
What is the physical host CPU model ?

Host CPU on this test machine is a i9-13900. Will check on the VBS topic.

 

[AllanM]

This is the cost of nested virtualization used by VBS.
Easier is switch vCPU type to x86-64-v2-aes or v3 if newer physical CPU.
What is the physical host CPU model ?

Unfortunately, this isn't a valid option for modern enterprise environments. VBS and associated technologies are effectively required by contractual obligation for many businesses these days. There are security controls and configuration baselines that require it.

My understanding is that modern CPUs have nested virtualization/paging capabilities that should allow this to work with very little performance compromise. It seems to me like there's just a implementation problem/bug in KVM/QEMU preventing it from working as intended.

 

[orange]

I went ahead and tried the following:
- Disabled nested virtualization using kvm-intel.conf, rebooted the server
- Set the virtual machine to use x86-64-v4 and x86-64-v2-AES

Unfortunately, that did not change the performance characteristics related to idle cpu interrupts on the Windows Server 2025 virtual machine.



msinfo32

 

[SleepyCircuit]

@orange Did you find a fix yet?

I am having the same issue with a Windows Server 2022 VM upgraded to Windows Server 2025 on an Intel Core i7-12700 host.
It is idling at about 2-3% CPU usage vs < 1% on WS2022 with 8 cores configured as host.

in powertop, on the host, Pkg(HW) never enters any C-state when WS2025 is running. With WS2022 running, it spends about 10-20% of it's time in C2 (idle).

VBS is disabled and the WS2025 VM itself is quite snappy via RDP.

 

[_gabriel]

configured as host

Have you tried as vCpu type x86-64-v2-aes ?

 

[SleepyCircuit]

Have you tried as vCpu type x86-64-v2-aes ?

Yes, I forgot to mention it, as it's been referenced earlier in the thread. I tried x86-64-v2-AES and x86-64-v3. Results were the same.

I did not however try to thinker with spec-ctl / pcid flags, as it is unclear to me if I should select minus (-) or plus (+) to have an effect.

 

[AlexanderT]

Just encountered the same issue after upgrading a Win 10 Pro VM to Win 11 Pro. VBS off and followed all advices from this thread. Still having idle percentage 2 to 3 times higher than before the upgrade. Inside the VM, there is virtually no CPU usage, except I am seeing more time spent with interrupts that it used to. Perhaps a virtio driver issue? Did you find any solution in the meantime?

 

[SleepyCircuit]

I haven't found a solution yet on my side. I spent quite some time on the issue and I was able to completely disable spectre mitigation on the host (pve), in the guest VM, disable power management, make sure vbs is disabled, etc but it did not make a significant improvement. It looks like the newer kernels handles idle cores differently.

Those are my host interrupts on Windows 2022. Note that only the first vCore generates events while the VM is idle.

Usage Events/s Description
6.1 ms/s 33.8 [PID 2616656] /usr/bin/kvm -id 116 -name vm-windows2022
7.0 ms/s 0.7 [PID 2616657] /usr/bin/kvm -id 116 -name vm-windows2022
5.9 ms/s 1.1 [PID 2616661] /usr/bin/kvm -id 116 -name vm-windows2022
4.2 ms/s 1.6 [PID 2616659] /usr/bin/kvm -id 116 -name vm-windows2022

Those are interrupts on Windows 2025. It is the same base VM which has been upgraded from Windows 2022 and is idle / up to date.

Usage Events/s Description
16.8 ms/s 221.1 [PID 2622138] /usr/bin/kvm -id 110 -name vm-windows2025
13.8 ms/s 201.0 [PID 2622140] /usr/bin/kvm -id 110 -name vm-windows2025
11.8 ms/s 180.3 [PID 2622139] /usr/bin/kvm -id 110 -name vm-windows2025
11.2 ms/s 130.5 [PID 2622137] /usr/bin/kvm -id 110 -name vm-windows2025

It seems the newer kernel generates interrupts on each cores whether they're idle or not.
I tried increasing the vCore count to 8 and they also each get about 100-200 events/s.

In the end, it increases the power consumption by 2W on my host, so I stopped chasing the issue.

 

[nodoame]

I can reproduce this on a plain Debian testing machine using fresh Windows 11 24H2 / 25H2. 23H2 / Windows 10 22H2 are unaffected.
Unfortunately, I can't just ignore this as not only does power consumption increase but those interrupts also prevent CPU cores from entering C3 state which in turn deactivates Intel Turbo Boost.

# 23H2 (cpuset=uneven cores)
$ perf kvm --host stat live
                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                             EOI_INDUCED        494    48.96%     0.03%      1.43us      6.00us      4.04us ( +-   0.72% )
                                     HLT        492    48.76%    98.96%   4530.68us  15769.42us  15435.74us ( +-   0.32% )
                           EPT_MISCONFIG          9     0.89%     1.02%     13.07us  77606.03us   8658.88us ( +-  99.53% )
                      EXTERNAL_INTERRUPT          5     0.50%     0.00%     14.94us     20.75us     17.80us ( +-   5.33% )
                                  VMCALL          5     0.50%     0.00%      4.66us     14.60us      8.37us ( +-  20.82% )
                              APIC_WRITE          4     0.40%     0.00%      5.20us      7.13us      6.41us ( +-   6.52% )

$ turbostat --quiet --interval 1
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     NMI     SMI     POLL%   C1%     C2%     C3%     CPU%c1  CPU%c6  CPU%c7  CoreTmp CoreThr PkgTmp  Totl%C0 Any%C0  GFX%C0  CPUGFX% Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc8 Pk%pc10 CPU%LPI SYS%LPI PkgWatt CorWatt GFXWatt RAMWatt PKG_%   RAM_%   UncMHz  SysWatt
-       -       13      1.64    813     2112    0.50    2892    0       0       0.00    1.05    6.96    90.47   3.33    38.03   55.69   39      0       40      37.05   22.30   0.00    0.00    57.25   0.00    0.00    0.00    0.00    0.00    0.00    2.36    1.10    0.00    0.00    0.00    0.00    800     0.00
0       0       11      1.43    800     2112    0.29    219     0       0       0.00    1.86    14.14   82.81   4.24    20.95   71.07   38      0       40      37.04   22.29   0.00    0.00    57.24   0.00    0.00    0.00    0.00    0.00    0.00    2.36    1.10    0.00    0.00    0.00    0.00    800
0       1       15      1.90    800     2112    0.34    200     0       0       0.00    1.34    10.45   86.56   4.23
4       2       13      1.68    801     2112    0.27    182     0       0       0.00    0.73    14.36   83.42   4.07    15.54   77.12   39      0
4       3       11      1.40    801     2112    0.27    140     0       0       0.00    0.35    7.95    90.50   4.07
8       4       17      1.97    859     2112    0.57    186     0       0       0.00    1.74    12.11   84.36   6.21    11.48   78.55   37      0
8       5       14      1.62    860     2112    0.64    128     0       0       0.00    0.33    4.39    93.82   6.21
12      6       11      1.41    801     2112    0.30    132     0       0       0.00    0.12    7.93    90.69   1.74    9.54    85.70   35      0
12      7       11      1.41    801     2112    0.29    122     0       0       0.00    0.25    4.95    93.55   1.74
16      8       6       0.75    806     2112    0.27    72      0       0       0.00    0.90    4.16    94.28   1.71    4.73    91.84   36      0
16      9       8       1.03    802     2112    0.31    98      0       0       0.00    0.26    2.19    96.62   1.71
20      10      5       0.60    803     2112    0.28    55      0       0       0.00    1.22    2.81    95.45   2.00    3.41    93.05   35      0
20      11      8       1.02    802     2112    0.32    96      0       0       0.00    0.37    2.09    96.61   2.00
24      12      15      1.72    867     2112    0.92    135     0       0       0.00    0.48    8.45    89.48   3.39    8.25    85.30   37      0
24      13      11      1.27    857     2112    0.28    112     0       0       0.00    0.17    4.69    94.00   3.39
28      14      12      1.45    805     2112    0.48    115     0       0       0.00    0.50    9.68    88.52   2.00    9.49    85.76   37      0
28      15      10      1.21    803     2112    0.30    115     0       0       0.00    0.18    4.29    94.47   2.00


# 24H2 (cpuset=uneven cores)
$ perf kvm --host stat live
                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                                     HLT       1105    49.46%    97.89%     20.43us  80025.30us   1827.12us ( +-   5.43% )
                             EOI_INDUCED       1099    49.19%     1.26%      0.64us  24351.12us     23.61us ( +-  93.83% )
                              APIC_WRITE         12     0.54%     0.00%      2.39us      4.90us      3.88us ( +-   6.18% )
                                  VMCALL          9     0.40%     0.00%      7.35us     16.19us     11.09us ( +-  10.53% )
                      EXTERNAL_INTERRUPT          7     0.31%     0.00%      4.24us     12.82us      6.83us ( +-  15.77% )
                           EPT_MISCONFIG          2     0.09%     0.84%     63.00us  17298.67us   8680.83us ( +-  99.27% )

$ turbostat --quiet --interval 1
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     NMI     SMI     POLL%   C1%     C2%     C3%     CPU%c1  CPU%c6  CPU%c7  CoreTmp CoreThr PkgTmp  Totl%C0 Any%C0  GFX%C0  CPUGFX% Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc8 Pk%pc10 CPU%LPI SYS%LPI PkgWatt CorWatt GFXWatt RAMWatt PKG_%   RAM_%   UncMHz  SysWatt
-       -       39      3.08    1277    2112    1.30    9082    0       0       0.01    2.27    44.87   49.88   78.11   29.66   0.00    39      0       43      64.32   35.65   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    5.53    4.04    0.00    0.00    0.00    0.00    800     0.00
0       0       17      1.73    958     2112    1.15    344     0       0       0.03    2.13    31.74   64.50   96.13   0.00    0.00    39      0       43      64.32   35.65   0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    5.53    4.04    0.00    0.00    0.00    0.00    800
0       1       23      2.84    811     2112    1.01    871     0       0       0.00    2.44    94.89   0.00    96.12
4       2       3       0.38    815     2112    0.70    101     0       0       0.00    2.64    3.22    93.78   97.82   0.00    0.00    39      0
4       3       19      2.36    803     2112    1.10    855     0       0       0.00    1.43    96.37   0.00    97.83
8       4       11      1.06    1045    2112    0.87    158     0       0       0.03    2.15    10.54   86.29   96.81   0.00    0.00    37      0
8       5       23      2.72    829     2112    1.40    851     0       0       0.00    1.38    96.07   0.00    96.81
12      6       5       0.52    962     2112    0.76    88      0       0       0.00    2.82    5.10    91.58   97.74   0.00    0.00    36      0
12      7       19      2.30    806     2112    1.10    851     0       0       0.00    1.74    96.13   0.00    97.74
16      8       8       0.76    983     2112    0.69    142     0       0       0.00    3.16    6.56    89.55   97.32   0.00    0.00    38      0
16      9       20      2.52    804     2112    1.04    850     0       0       0.03    1.86    95.79   0.00    97.32
20      10      4       0.50    849     2112    0.91    203     0       0       0.00    6.70    4.13    88.71   97.58   0.00    0.00    38      0
20      11      20      2.48    806     2112    1.08    850     0       0       0.00    1.87    95.81   0.00    97.58
24      12      31      1.96    1580    2112    1.46    228     0       0       0.00    7.63    13.06   77.40   96.18   0.00    0.00    37      0
24      13      21      2.50    833     2112    1.01    849     0       0       0.03    1.57    96.10   0.00    96.18
28      14      9       1.03    915     2112    0.87    121     0       0       0.01    2.90    14.19   81.93   97.14   0.00    0.00    39      0
28      15      20      2.44    805     2112    1.05    851     0       0       0.03    1.47    96.26   0.00    97.14

Here I'm assigning the VM to all uneven cores which for some reason decreases C3 time from about 90% to exactly 0, despite CPU load only increasing by 1% or so per core on 24H2. Due to how Turbo Boost works this basically reduces single thread performance for the whole host with every additional vCPU on a 24H2+ VM.

I've tried a bunch of things like different CPU models, flags, enlightments, GlobalTimerResolutionRequests, bcdedit flags, disabling nested virtualization etc. but without much success.
The only way I found C3 to work is by enabling useplatformticks with HPET enabled. Unfortunately this results in even more interrupts and even larger CPU consumption in idle. This makes me believe something related to timer resolution changed between 23H2 and 24H2 but I'm out of ideas what else to try...