Windows VBS / HVCI / Credential Guard on PVE 9.2 (AMD EPYC and Intel Xeon): test results, needed VBS how-to or Proxmox team attention

Leonid Kanareykin

New Member
Sep 5, 2025
15
1
3
Hi all,

we migrated about 100 VMs from VMware to Proxmox VE and plan to move about 300 more Windows 11 Enterprise VDI desktops. Our security baseline requires VBS, HVCI (Memory Integrity) and Credential Guard. After the migration, Windows guests with VBS were slow, so we tested systematically. I am sharing the results because they may help others, and because we ended up with questions for the Proxmox team.

Environment
  • Proxmox VE 9.2.21, kernel 7.0.14-pve, QEMU 11.0.3 (pve-qemu-kvm 11.0.3-4)
  • Cluster 1 (most tests below): 5 nodes, Dell, AMD EPYC 7502P (Zen 2), Ceph RBD, HA
  • Cluster 2 (see "Intel cluster" below): 6 nodes of Dell, 4x Intel Xeon Platinum 8280 per node (Cascade Lake), stretched Ceph, same PVE/kernel/QEMU
  • AMD test guest: Windows 11 Pro 25H2 (26200), 4 vCPU, 8 GB, OVMF, Secure Boot (ms-cert=2023k, pre-enrolled-keys=1), vTPM 2.0, virtio-scsi-single + iothread
  • VBS enabled by registry: EnableVirtualizationBasedSecurity=1, RequirePlatformSecurityFeatures=1, HVCI Enabled=1

Method
A PowerShell script in the guest, started at least 4 minutes after boot:
  • Win32_DeviceGuard (available properties, VBS status, running services)
  • idle counters over 30 s (% Interrupt Time)
  • 100 x "cmd /c exit" (process spawn), median of 3
  • 2000 small file writes, median of 3
  • sha256 and gzip of 64 MB (pure CPU)
Percentages are relative to the same VM with VBS off (cpu: host).

Results (AMD cluster)
CPU configurationVBS + HVCI runningIdle interrupt %Spawn 100 procs2000 file writesLive migration
host, VBS off (baseline)-0.401397 ms1748 msyes
x86-64-v3no (no SLAT/virt exposed)---yes
hostno (property 1 "base virtualization" missing)---yes
host,flags=+nested-virtyes3.74+52%+92%yes, 65 ms downtime
same, machine pc-q35-11.0+pve2yes4.12+55%+95%yes
custom v1: + hv-tlbflush, hv-tlbflush-direct, hv-tlbflush-ext, hv-emsr-bitmap, hv-xmm-inputyes3.32+49%+92%yes
v2: v1 + hv-stimer-directyes3.97+103%+152%yes
v2-avic: v2 + hv-avicyes3.82+56%+103%yes
v3c: v2-avic + hv-frequencies, hv-crashyes3.32+46%+86%not tested
v3b: v3c + hv-reenlightenmentyes0.77+22%+14%NO - VM is lost
v3b + invtscguest hangs at boot----
v3b + tsc-frequency=2495312000 (host value)guest hangs at boot----
args: -cpu host,hv-passthrough (second VM)yes0.82+9% (own baseline)+26% (own baseline)no, blocked by PVE

Pure CPU work (sha256, gzip) is the same in every row. The whole cost is in kernel transitions: interrupts, process creation, file I/O.

Findings (AMD cluster)

1. On AMD, a single flag makes the difference: hv-reenlightenment.

Rows v3c and v3b differ only by this flag. Proxmox VE never sets it. This is the fast model:
Code:
cpu-model: win-vbs-v3b
    flags +svm;+hv-frequencies;+hv-reenlightenment;+hv-crash;+hv-stimer-direct;+hv-tlbflush;+hv-tlbflush-direct;+hv-tlbflush-ext;+hv-emsr-bitmap;+hv-xmm-input;+hv-avic
    phys-bits host
    hidden 0
    reported-model host

2. Live migration with hv-reenlightenment kills the VM, and PVE does not prevent it.
The source reports success, then the target QEMU aborts:
Code:
migration completed, transferred 2.0 GiB VM-state
average migration speed: 4.0 GiB/s - downtime 86 ms
ERROR: tunnel replied 'ERR: resume failed - VM 106 qmp command 'query-status' failed - client closed connection' to command 'resume 106'
ERROR: migration finished with problems

target node journal:
kvm: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
kvm: load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'cpu': Loading VM subsection 'cpu/msr_hyperv_reenlightenment' in 'cpu' failed: -22: post load hook failed
The VM ends up stopped on the target. With hv-passthrough, PVE at least refuses the migration up front.

3. The fix QEMU asks for hangs the guest.
Both nodes report the same TSC (2495.312 MHz). With args: -global host-x86_64-cpu.tsc-frequency=2495312000, or with +invtsc in the model, QEMU starts without any warning, and the guest hangs right after "BdsDxe: starting Boot0004 Windows Boot Manager": one vCPU at 100%, no disk I/O. Without VBS-related flags we did not test this, so it may be specific to nested SVM plus the Windows hypervisor.

So on AMD we found no configuration that is both fast and live-migratable.

4. cpu: host alone does not expose SVM.
VBS only starts with flags=+nested-virt. The generated command line for a custom model with +nested-virt and +pve2:
Code:
-cpu 'host,-cet-ibt,-cet-ss,+hv-avic,+hv-emsr-bitmap,+hv-stimer-direct,+hv-tlbflush,+hv-tlbflush-direct,+hv-tlbflush-ext,+hv-xmm-input,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+svm,host-phys-bits=true'

5. pc-q35-11.0+pve2 has no effect on AMD.
No hv-evmcs in the command line (expected, it is Intel-only), and "running-machine" reports pc-q35-11.0+pve0. There is no AMD counterpart.

6. CET is always masked (-cet-ibt,-cet-ss), so Windows kernel-mode hardware-enforced stack protection will not be available on CET-capable hosts.

7. Security properties that never appear in Win32_DeviceGuard.AvailableSecurityProperties: 3 (DMA protection), 6 (SMM security mitigations), 8 (APIC virtualization). We get 1,2,4,5,7. Our existing GPO requires Secure Boot + DMA protection (RequirePlatformSecurityFeatures=3). With it, VBS stays "configured but not running", so we had to weaken the policy.

8. Custom CPU models and args need root@pam. Not workable for hundreds of VMs with delegated permissions.

Intel cluster (Xeon Platinum 8280): production experience

A colleague enabled VBS + Credential Guard on 5 production machines there (3 Windows 11 Enterprise VDI desktops, Windows Server 2019 and 2022). This is the configuration that works for us today:
Code:
machine: pc-q35-11.0+pve2
cpu: host
args: -cpu host,-cet-ibt,-cet-ss,-vmx-mbec,+hv-evmcs,+hv-tlbflush,+hv-tlbflush-direct,+hv-tlbflush-ext,+hv-emsr-bitmap,+hv-xmm-input,+hv-reenlightenment,+hv-crash,hv-frequencies,hv-stimer-direct,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt
Guest policy: Platform Security Level = Secure Boot (no DMA), Credential Guard enabled without UEFI lock, HVCI explicitly Disabled.

Measurements there (8 vCPU VDI guest; taken while hv-apicv was still in the list, see point 2 below, so they need to be repeated):
ConfigurationHost cores used by an idle guestProgram start, worst case
no VBS0.26baseline
VBS, nested virt + two hv flags1.23-32%
VBS, eight hv flags, no hv-apicv1.02-27%
VBS, full explicit list with hv-apicv0.44-6.4%

What we hit on Intel:
  1. Guest freezes for 3-6 minutes with all vCPUs at 100% (kernel 7.0, Windows with VBS). We had 29 minutes of such freezes in 2.5 hours on one VDI desktop. We believe this is Proxmox bug 7825. Workaround that stopped it completely: -vmx-mbec plus machine type pc-q35-11.0+pve2.
  2. hv-apicv made the freezes worse, so we removed it. It was also the flag with the largest performance gain (table above), so we lost most of that gain.
  3. HVCI is not usable for us on Intel: the host shows 300-400% CPU for the VM while Windows looks idle. We run Credential Guard only. "Not configured" is not enough on current Windows 11 builds, HVCI must be set to Disabled explicitly. Note that the workaround in point 1 disables MBEC, which HVCI relies on for acceptable speed.
  4. viommu=intel does not give Kernel DMA Protection: Windows boots into the recovery environment. So the "Secure Boot and DMA Protection" level cannot be satisfied.
  5. The EFI vars disk remembers the old VBS lock. A VM that came from VMware with a policy "Secure Boot + DMA, UEFI lock" keeps that requirement in efidisk0, and VBS stays at status 1 (event 153). We had to fix the registry first (Locked=0, RequirePlatformSecurityFeatures=1) and only then recreate efidisk0. With BitLocker, suspend it before replacing efidisk0.
  6. Live migration is not tested yet on Intel. The args line contains hv-reenlightenment without tsc-frequency, so after the AMD result above we expect the same failure. We will not live-migrate these VMs until this is clarified.

Questions to the Proxmox team
  1. Is the boot hang with tsc-frequency= / invtsc under nested SVM a known KVM/QEMU issue? Is there a working way to combine hv-reenlightenment with live migration on AMD?
  2. Can PVE block migration of a guest that uses hv-reenlightenment without tsc-frequency, instead of losing the VM?
  3. Are there plans for a built-in, migratable "Windows VBS" CPU option with an explicit hv-* set, selectable in the GUI/API without root?
  4. What is the recommended configuration today for VBS guests on AMD Zen 2 and on Intel Cascade Lake that keeps live migration and HA?
  5. Is there an AMD counterpart to the +pve2 machine fix?
  6. Intel: what is the status of bug 7825 (freezes with VBS on kernel 7.0)? When it is fixed, can MBEC and hv-apicv be enabled again, so that HVCI becomes usable?
  7. Are a virtual IOMMU accepted by Windows for Kernel DMA Protection, SMM mitigations in OVMF, and guest CET on the roadmap?

Related threads:

We can test patches or pre-release packages on our test nodes and share the benchmark script. Thanks!
 
Last edited:
Hello @Leonid Kanareykin

thank you for posting on the Proxmox Forum and for your detailed report!

If you could post your benchmarking script i can try to reproduce your findings and work to improve them if possible.

Is the boot hang with tsc-frequency= / invtsc under nested SVM a known KVM/QEMU issue? Is there a working way to combine hv-reenlightenment with live migration on AMD?
I was not able to reproduce this issue locally on a Zen 5 based AMD CPU.
The Windows VM booted fine with CPU model host and a fixed tsc-frequency.
Your args seem to set the frequency in the global parameter while it is a -cpu parameter so maybe try setting it there.

Currently no offical way exists, but i am looking into it.
Technically Intel CPUs from the Xeon Scalaable family and AMD EPYC CPUs should be able to live migrate with hv-reenlightenment enabled since both of them seem to support TSC scaling, but i have not yet tested that.

Can PVE block migration of a guest that uses hv-reenlightenment without tsc-frequency, instead of losing the VM?
Currently there is no way of checking this before migrating, but i am looking into a pre migration check for that. There are also related bugreports [0].

Are there plans for a built-in, migratable "Windows VBS" CPU option with an explicit hv-* set, selectable in the GUI/API without root?
Currently there is no such plan.
If you have a specific feature request in mind, please feel free to open a bug report at [1] and state your issue and use case there.

What is the recommended configuration today for VBS guests on AMD Zen 2 and on Intel Cascade Lake that keeps live migration and HA?
There is no specific recommendation for VBS VMs right now.
In general this article [2] provides some guidance on some performance tweaks for Windows Server 2025.

Is there an AMD counterpart to the +pve2 machine fix?
The fix introduced by 11.0+pve2 was specific for an issue [3] for Intel CPUs and did not affect AMD CPUs at the time, so no there is no AMD counterpart.

Intel: what is the status of bug 7825 (freezes with VBS on kernel 7.0)? When it is fixed, can MBEC and hv-apicv be enabled again, so that HVCI becomes usable?
The observed behavior seems to be related to an underlying kernel issue as reported here [4] and there is currently no known fix or workaround for that.
As mentioned in the Bugzilla entry the main issue seems to be the combination of APICv and MBEC. The latter by itself does not seem to have issues on Intel CPUs and such was kept enabled with 11.0+pve2 since it already provides some performance benefits.

Best regards
Jonas Theisen

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=2924
[1] https://bugzilla.proxmox.com/
[2] https://pve.proxmox.com/wiki/Windows_2025_guest_best_practices
[3] https://pve.proxmox.com/wiki/Roadma...ype and VBS enabled may intermittently freeze
[4] https://lore.kernel.org/all/bdb4a6dc-1d81-4cb4-802f-f9638009d6c5@proxmox.com/
 
Hi Jonas,

thank you for the quick and detailed answer, and for looking into the pre-migration check.

Benchmark script
Run as Administrator in the guest: powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:\vbsbench.ps1. It waits until 4 minutes of uptime, then takes about 5-6 minutes and prints one line of JSON. We compare runs on the same VM with VBS off and on (VBS on = EnableVirtualizationBasedSecurity=1, RequirePlatformSecurityFeatures=1, HVCI Enabled=1).
Note: in the numbers from my first post the field "cg" was mislabeled, it is CodeIntegrityPolicyEnforcementStatus (renamed to ci_policy below).

Code:
# vbsbench.ps1 - micro-benchmark for Windows guests with/without VBS.
# Run as Administrator inside the guest:
#   powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:\vbsbench.ps1
# Output: one line of JSON. Takes about 5-6 minutes, plus a wait until 4 minutes of uptime.
# Compare runs on the same VM (same vCPU/RAM/storage), VBS off vs. VBS on.

$ErrorActionPreference = 'SilentlyContinue'
$r = [ordered]@{}

# Wait until the guest is at least 4 minutes up (boot-time Defender/indexing noise).
$up = ((Get-Date) - (Get-CimInstance Win32_OperatingSystem).LastBootUpTime).TotalSeconds
if ($up -lt 240) { Start-Sleep -Seconds ([int](240 - $up)) }

# VBS state.
#   avail: AvailableSecurityProperties (1 base virt, 2 Secure Boot, 3 DMA, 4 Secure MOR,
#          5 NX/MAT, 6 SMM mitigations, 7 MBEC/GMET, 8 APIC virtualization)
#   vbs:   VirtualizationBasedSecurityStatus (0 off, 1 configured not running, 2 running)
#   run:   SecurityServicesRunning (1 Credential Guard, 2 HVCI)
$d = Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard -ClassName Win32_DeviceGuard
$r.avail = $d.AvailableSecurityProperties -join ','
$r.req   = $d.RequiredSecurityProperties -join ','
$r.vbs   = $d.VirtualizationBasedSecurityStatus
$r.conf  = $d.SecurityServicesConfigured -join ','
$r.run   = $d.SecurityServicesRunning -join ','
$r.ci_policy = $d.CodeIntegrityPolicyEnforcementStatus

# Idle counters: 15 samples x 2 s, averaged.
$c = Get-Counter '\Processor(_Total)\% Processor Time','\Processor(_Total)\% Interrupt Time','\Processor(_Total)\% Privileged Time','\Processor(_Total)\Interrupts/sec','\System\Context Switches/sec' -SampleInterval 2 -MaxSamples 15
$g = $c.CounterSamples | Group-Object { ($_.Path -split '\\')[-1] }
foreach ($x in $g) { $r['idle_' + $x.Name] = [math]::Round(($x.Group | Measure-Object CookedValue -Average).Average, 2) }

# Pure CPU: SHA-256 of 64 MB x 8, gzip of 64 MB x 2.
$b = New-Object byte[] (64MB); (New-Object Random 1).NextBytes($b)
$sha = [Security.Cryptography.SHA256]::Create()
$r.sha_ms  = [int](Measure-Command { 1..8 | % { [void]$sha.ComputeHash($b) } }).TotalMilliseconds
$r.gzip_ms = [int](Measure-Command { 1..2 | % { $ms = New-Object IO.MemoryStream; $gz = New-Object IO.Compression.GZipStream($ms, [IO.Compression.CompressionLevel]::Optimal); $gz.Write($b, 0, $b.Length); $gz.Close() } }).TotalMilliseconds

# Process creation: 100 x "cmd /c exit", median of 3.
$s = @(); 1..3 | % { $s += [int](Measure-Command { 1..100 | % { cmd /c exit } }).TotalMilliseconds }
$r.spawn100_ms = ($s | Sort)[1]

# Small file I/O: 2000 tiny files written and deleted, median of 3.
$p = "$env:TEMP\vb"; mkdir $p -Force | Out-Null
$f = @(); 1..3 | % { $f += [int](Measure-Command { 1..2000 | % { [IO.File]::WriteAllText("$p\$_.txt", 'x') }; Get-ChildItem $p | Remove-Item -Force }).TotalMilliseconds }
$r.file2000_ms = ($f | Sort)[1]

# Memory: touch every page of a 512 MB array, median of 3.
$m = @(); 1..3 | % { $m += [int](Measure-Command { $a = New-Object byte[] (512MB); for ($i = 0; $i -lt $a.Length; $i += 4096) { $a[$i] = 1 }; $a = $null; [GC]::Collect() }).TotalMilliseconds }
$r.memtouch512_ms = ($m | Sort)[1]

# Scheduler: 20 thread create/join + 20000 Sleep(0), median of 3.
$t = @(); 1..3 | % { $t += [int](Measure-Command { 1..20 | % { $th = [Threading.Thread]::new({}); $th.Start(); $th.Join() }; 1..20000 | % { [Threading.Thread]::Sleep(0) } }).TotalMilliseconds }
$r.sched_ms = ($t | Sort)[1]

$r | ConvertTo-Json -Compress

tsc-frequency
You are right, the first time I passed it as -global host-x86_64-cpu.tsc-frequency=2495312000. I repeated the test with tsc-frequency as a -cpu property. The result is the same: the guest hangs right after "BdsDxe: starting Boot0004 Windows Boot Manager".

Config:
Code:
machine: pc-q35-11.0+pve2
cpu: host
args: -cpu host,-cet-ibt,-cet-ss,+svm,+hv-avic,+hv-emsr-bitmap,+hv-stimer-direct,+hv-tlbflush,+hv-tlbflush-direct,+hv-tlbflush-ext,+hv-xmm-input,+hv-frequencies,+hv-reenlightenment,+hv-crash,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,host-phys-bits=true,tsc-frequency=2495312000
The exact same line without tsc-frequency= boots fine, with VBS running. Adding +invtsc instead of tsc-frequency hangs the same way.

Host: AMD EPYC 7502P (Zen 2), kernel 7.0.14-25-pve, pve-qemu-kvm 11.0.3-4. Both nodes report "tsc: Refined TSC clocksource calibration: 2495.31x MHz". Guest: Windows 11 Pro 25H2 (26200), VBS + HVCI enabled, OVMF, Secure Boot, vTPM.

The host kernel log shows one difference between a hang and a good boot:
Code:
hang (only this line, nothing after it):
kvm_amd: kvm [793760]: vcpu0, guest rIP: 0xfffff804f3b3e249 Unhandled WRMSR(0xc0010115) = 0x0

good boot (same flags, no tsc-frequency):
kvm_amd: kvm [523422]: vcpu0, guest rIP: 0xfffff8276a13e249 Unhandled WRMSR(0xc0010115) = 0x0
kvm_amd: kvm [523422]: vcpu1, guest rIP: 0xfffff8276a13e249 Unhandled WRMSR(0xc0010115) = 0x0
kvm_amd: kvm [523422]: vcpu2, guest rIP: 0xfffff8276a13e249 Unhandled WRMSR(0xc0010115) = 0x0
kvm_amd: kvm [523422]: vcpu3, guest rIP: 0xfffff8276a13e249 Unhandled WRMSR(0xc0010115) = 0x0
So the WRMSR itself seems harmless. With a fixed TSC frequency the guest stops on vcpu0 right after it, before the other vCPUs are brought up. Since you could not reproduce on Zen 5, this may be specific to Zen 2.

To rule out VBS, I disabled the Windows hypervisor in the guest (bcdedit /set {current} hypervisorlaunchtype off) and booted with the same tsc-frequency line. It still hangs, with the same single vcpu0 log line. So the hang is not caused by nested Hyper-V: the Windows kernel itself stops during early boot on Zen 2 when the TSC frequency is fixed (or with invtsc). Host microcode is 0x830107d.

If it helps, I can try a minimal line without any of the extra hv-* flags (only the Proxmox defaults plus tsc-frequency), or collect a trace of the guest, just tell me what you need.

Intel / bug 7825
Thank you for the clarification that the problem is the combination of APICv and MBEC, and that MBEC alone is fine. On our Intel nodes we had disabled MBEC (-vmx-mbec) in addition to pc-q35-11.0+pve2. We will test with MBEC enabled again and only hv-apicv removed.

Feature request
We will open a Bugzilla entry for a migratable Windows VBS CPU option (explicit hv-* set, including hv-reenlightenment with tsc-frequency) selectable without root, and will link it here.

Best regards,
Leonid