Hi all,
we migrated about 100 VMs from VMware to Proxmox VE and plan to move about 300 more Windows 11 Enterprise VDI desktops. Our security baseline requires VBS, HVCI (Memory Integrity) and Credential Guard. After the migration, Windows guests with VBS were slow, so we tested systematically. I am sharing the results because they may help others, and because we ended up with questions for the Proxmox team.
Environment
Method
A PowerShell script in the guest, started at least 4 minutes after boot:
Results (AMD cluster)
Pure CPU work (sha256, gzip) is the same in every row. The whole cost is in kernel transitions: interrupts, process creation, file I/O.
Findings (AMD cluster)
1. On AMD, a single flag makes the difference: hv-reenlightenment.
Rows v3c and v3b differ only by this flag. Proxmox VE never sets it. This is the fast model:
2. Live migration with hv-reenlightenment kills the VM, and PVE does not prevent it.
The source reports success, then the target QEMU aborts:
The VM ends up stopped on the target. With hv-passthrough, PVE at least refuses the migration up front.
3. The fix QEMU asks for hangs the guest.
Both nodes report the same TSC (2495.312 MHz). With
So on AMD we found no configuration that is both fast and live-migratable.
4. cpu: host alone does not expose SVM.
VBS only starts with flags=+nested-virt. The generated command line for a custom model with +nested-virt and +pve2:
5. pc-q35-11.0+pve2 has no effect on AMD.
No hv-evmcs in the command line (expected, it is Intel-only), and "running-machine" reports pc-q35-11.0+pve0. There is no AMD counterpart.
6. CET is always masked (-cet-ibt,-cet-ss), so Windows kernel-mode hardware-enforced stack protection will not be available on CET-capable hosts.
7. Security properties that never appear in Win32_DeviceGuard.AvailableSecurityProperties: 3 (DMA protection), 6 (SMM security mitigations), 8 (APIC virtualization). We get 1,2,4,5,7. Our existing GPO requires Secure Boot + DMA protection (RequirePlatformSecurityFeatures=3). With it, VBS stays "configured but not running", so we had to weaken the policy.
8. Custom CPU models and args need root@pam. Not workable for hundreds of VMs with delegated permissions.
Intel cluster (Xeon Platinum 8280): production experience
A colleague enabled VBS + Credential Guard on 5 production machines there (3 Windows 11 Enterprise VDI desktops, Windows Server 2019 and 2022). This is the configuration that works for us today:
Guest policy: Platform Security Level = Secure Boot (no DMA), Credential Guard enabled without UEFI lock, HVCI explicitly Disabled.
Measurements there (8 vCPU VDI guest; taken while hv-apicv was still in the list, see point 2 below, so they need to be repeated):
What we hit on Intel:
Questions to the Proxmox team
Related threads:
We can test patches or pre-release packages on our test nodes and share the benchmark script. Thanks!
we migrated about 100 VMs from VMware to Proxmox VE and plan to move about 300 more Windows 11 Enterprise VDI desktops. Our security baseline requires VBS, HVCI (Memory Integrity) and Credential Guard. After the migration, Windows guests with VBS were slow, so we tested systematically. I am sharing the results because they may help others, and because we ended up with questions for the Proxmox team.
Environment
- Proxmox VE 9.2.21, kernel 7.0.14-pve, QEMU 11.0.3 (pve-qemu-kvm 11.0.3-4)
- Cluster 1 (most tests below): 5 nodes, Dell, AMD EPYC 7502P (Zen 2), Ceph RBD, HA
- Cluster 2 (see "Intel cluster" below): Dell, 4x Intel Xeon Platinum 8280 per node (Cascade Lake), stretched Ceph, same PVE/kernel/QEMU
- AMD test guest: Windows 11 Pro 25H2 (26200), 4 vCPU, 8 GB, OVMF, Secure Boot (ms-cert=2023k, pre-enrolled-keys=1), vTPM 2.0, virtio-scsi-single + iothread
- VBS enabled by registry: EnableVirtualizationBasedSecurity=1, RequirePlatformSecurityFeatures=1, HVCI Enabled=1
Method
A PowerShell script in the guest, started at least 4 minutes after boot:
- Win32_DeviceGuard (available properties, VBS status, running services)
- idle counters over 30 s (% Interrupt Time)
- 100 x "cmd /c exit" (process spawn), median of 3
- 2000 small file writes, median of 3
- sha256 and gzip of 64 MB (pure CPU)
Results (AMD cluster)
| CPU configuration | VBS + HVCI running | Idle interrupt % | Spawn 100 procs | 2000 file writes | Live migration |
|---|---|---|---|---|---|
| host, VBS off (baseline) | - | 0.40 | 1397 ms | 1748 ms | yes |
| x86-64-v3 | no (no SLAT/virt exposed) | - | - | - | yes |
| host | no (property 1 "base virtualization" missing) | - | - | - | yes |
| host,flags=+nested-virt | yes | 3.74 | +52% | +92% | yes, 65 ms downtime |
| same, machine pc-q35-11.0+pve2 | yes | 4.12 | +55% | +95% | yes |
| custom v1: + hv-tlbflush, hv-tlbflush-direct, hv-tlbflush-ext, hv-emsr-bitmap, hv-xmm-input | yes | 3.32 | +49% | +92% | yes |
| v2: v1 + hv-stimer-direct | yes | 3.97 | +103% | +152% | yes |
| v2-avic: v2 + hv-avic | yes | 3.82 | +56% | +103% | yes |
| v3c: v2-avic + hv-frequencies, hv-crash | yes | 3.32 | +46% | +86% | not tested |
| v3b: v3c + hv-reenlightenment | yes | 0.77 | +22% | +14% | NO - VM is lost |
| v3b + invtsc | guest hangs at boot | - | - | - | - |
| v3b + tsc-frequency=2495312000 (host value) | guest hangs at boot | - | - | - | - |
| args: -cpu host,hv-passthrough (second VM) | yes | 0.82 | +9% (own baseline) | +26% (own baseline) | no, blocked by PVE |
Pure CPU work (sha256, gzip) is the same in every row. The whole cost is in kernel transitions: interrupts, process creation, file I/O.
Findings (AMD cluster)
1. On AMD, a single flag makes the difference: hv-reenlightenment.
Rows v3c and v3b differ only by this flag. Proxmox VE never sets it. This is the fast model:
Code:
cpu-model: win-vbs-v3b
flags +svm;+hv-frequencies;+hv-reenlightenment;+hv-crash;+hv-stimer-direct;+hv-tlbflush;+hv-tlbflush-direct;+hv-tlbflush-ext;+hv-emsr-bitmap;+hv-xmm-input;+hv-avic
phys-bits host
hidden 0
reported-model host2. Live migration with hv-reenlightenment kills the VM, and PVE does not prevent it.
The source reports success, then the target QEMU aborts:
Code:
migration completed, transferred 2.0 GiB VM-state
average migration speed: 4.0 GiB/s - downtime 86 ms
ERROR: tunnel replied 'ERR: resume failed - VM 106 qmp command 'query-status' failed - client closed connection' to command 'resume 106'
ERROR: migration finished with problems
target node journal:
kvm: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
kvm: load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'cpu': Loading VM subsection 'cpu/msr_hyperv_reenlightenment' in 'cpu' failed: -22: post load hook failed3. The fix QEMU asks for hangs the guest.
Both nodes report the same TSC (2495.312 MHz). With
args: -global host-x86_64-cpu.tsc-frequency=2495312000, or with +invtsc in the model, QEMU starts without any warning, and the guest hangs right after "BdsDxe: starting Boot0004 Windows Boot Manager": one vCPU at 100%, no disk I/O. Without VBS-related flags we did not test this, so it may be specific to nested SVM plus the Windows hypervisor.So on AMD we found no configuration that is both fast and live-migratable.
4. cpu: host alone does not expose SVM.
VBS only starts with flags=+nested-virt. The generated command line for a custom model with +nested-virt and +pve2:
Code:
-cpu 'host,-cet-ibt,-cet-ss,+hv-avic,+hv-emsr-bitmap,+hv-stimer-direct,+hv-tlbflush,+hv-tlbflush-direct,+hv-tlbflush-ext,+hv-xmm-input,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+svm,host-phys-bits=true'5. pc-q35-11.0+pve2 has no effect on AMD.
No hv-evmcs in the command line (expected, it is Intel-only), and "running-machine" reports pc-q35-11.0+pve0. There is no AMD counterpart.
6. CET is always masked (-cet-ibt,-cet-ss), so Windows kernel-mode hardware-enforced stack protection will not be available on CET-capable hosts.
7. Security properties that never appear in Win32_DeviceGuard.AvailableSecurityProperties: 3 (DMA protection), 6 (SMM security mitigations), 8 (APIC virtualization). We get 1,2,4,5,7. Our existing GPO requires Secure Boot + DMA protection (RequirePlatformSecurityFeatures=3). With it, VBS stays "configured but not running", so we had to weaken the policy.
8. Custom CPU models and args need root@pam. Not workable for hundreds of VMs with delegated permissions.
Intel cluster (Xeon Platinum 8280): production experience
A colleague enabled VBS + Credential Guard on 5 production machines there (3 Windows 11 Enterprise VDI desktops, Windows Server 2019 and 2022). This is the configuration that works for us today:
Code:
machine: pc-q35-11.0+pve2
cpu: host
args: -cpu host,-cet-ibt,-cet-ss,-vmx-mbec,+hv-evmcs,+hv-tlbflush,+hv-tlbflush-direct,+hv-tlbflush-ext,+hv-emsr-bitmap,+hv-xmm-input,+hv-reenlightenment,+hv-crash,hv-frequencies,hv-stimer-direct,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhaltMeasurements there (8 vCPU VDI guest; taken while hv-apicv was still in the list, see point 2 below, so they need to be repeated):
| Configuration | Host cores used by an idle guest | Program start, worst case |
|---|---|---|
| no VBS | 0.26 | baseline |
| VBS, nested virt + two hv flags | 1.23 | -32% |
| VBS, eight hv flags, no hv-apicv | 1.02 | -27% |
| VBS, full explicit list with hv-apicv | 0.44 | -6.4% |
What we hit on Intel:
- Guest freezes for 3-6 minutes with all vCPUs at 100% (kernel 7.0, Windows with VBS). We had 29 minutes of such freezes in 2.5 hours on one VDI desktop. We believe this is Proxmox bug 7825. Workaround that stopped it completely:
-vmx-mbecplus machine type pc-q35-11.0+pve2. - hv-apicv made the freezes worse, so we removed it. It was also the flag with the largest performance gain (table above), so we lost most of that gain.
- HVCI is not usable for us on Intel: the host shows 300-400% CPU for the VM while Windows looks idle. We run Credential Guard only. "Not configured" is not enough on current Windows 11 builds, HVCI must be set to Disabled explicitly. Note that the workaround in point 1 disables MBEC, which HVCI relies on for acceptable speed.
- viommu=intel does not give Kernel DMA Protection: Windows boots into the recovery environment. So the "Secure Boot and DMA Protection" level cannot be satisfied.
- The EFI vars disk remembers the old VBS lock. A VM that came from VMware with a policy "Secure Boot + DMA, UEFI lock" keeps that requirement in efidisk0, and VBS stays at status 1 (event 153). We had to fix the registry first (Locked=0, RequirePlatformSecurityFeatures=1) and only then recreate efidisk0. With BitLocker, suspend it before replacing efidisk0.
- Live migration is not tested yet on Intel. The args line contains hv-reenlightenment without tsc-frequency, so after the AMD result above we expect the same failure. We will not live-migrate these VMs until this is clarified.
Questions to the Proxmox team
- Is the boot hang with tsc-frequency= / invtsc under nested SVM a known KVM/QEMU issue? Is there a working way to combine hv-reenlightenment with live migration on AMD?
- Can PVE block migration of a guest that uses hv-reenlightenment without tsc-frequency, instead of losing the VM?
- Are there plans for a built-in, migratable "Windows VBS" CPU option with an explicit hv-* set, selectable in the GUI/API without root?
- What is the recommended configuration today for VBS guests on AMD Zen 2 and on Intel Cascade Lake that keeps live migration and HA?
- Is there an AMD counterpart to the +pve2 machine fix?
- Intel: what is the status of bug 7825 (freezes with VBS on kernel 7.0)? When it is fixed, can MBEC and hv-apicv be enabled again, so that HVCI becomes usable?
- Are a virtual IOMMU accepted by Windows for Kernel DMA Protection, SMM mitigations in OVMF, and guest CET on the roadmap?
Related threads:
- https://forum.proxmox.com/threads/proxmox-ve-8-4-14-and-virtualization-based-security-vbs.174821
- https://forum.proxmox.com/threads/v...-does-it-require-nested-virtualization.168788
- https://forum.proxmox.com/threads/t...e-of-windows-when-the-cpu-type-is-host.163114
- https://forum.proxmox.com/threads/my-rds-2025-vm-running-extremely-slow.177352
- QEMU Hyper-V enlightenments: https://www.qemu.org/docs/master/system/i386/hyperv.html
We can test patches or pre-release packages on our test nodes and share the benchmark script. Thanks!
Last edited: