Win Server 2025 VM: DPC_WATCHDOG_VIOLATION 0x133 after upgrade to Kernel 7.x

[pmbaeum]

I recently ran into a problem that gave me quite a headache, and since I haven't fully resolved it yet, I wanted to share my findings here and ask if anyone has seen something similar.


Environment
PVE9.2.3, Kernel 7.0.2-7-pve (since 02.06.2026), Windows Server 2025 VM running Exchange SE. VM config: q35, cpu: host, 12 vCores, 64GB RAM, virtio-scsi-single with multiple disks.


Symptom
After the host was running Kernel 7.0.2-7, the Windows Server 2025 VM started crashing with DPC_WATCHDOG_VIOLATION (0x133) consistently about 2 minutes after boot.
An additional symptom: directly after VM boot, the system clock was off by -2 hours, despite NTP being active.
In Safe-Mode the VM runs just fine.

Dump analysis
Two usable kernel dumps were captured via WinDbg. Both pointed to the same root cause:

# Dump 1:
FAILURE_BUCKET_ID: 0x133_OUT_OF_SYNC_PROCESSOR_nt!HvlpGetRegister64

Stack:
  nt!HvlpGetRegister64
  nt!KeQueryPerformanceCounter
  nt!KiSetSystemTimeDpc → nt!KiUpdateTime → nt!KeAccumulateTicks
  nt!KeBugCheckEx

# Dump 2:
FAILURE_BUCKET_ID: 0x133_OUT_OF_SYNC_PROCESSOR_nt!KiIpiStallOnPacketTargetsPrcb
PROCESS_NAME: w3wp.exe

Stack:
  nt!KiIpiStallOnPacketTargetsPrcb
  nt!KeFlushProcessWriteBuffers
  nt!ExpGetProcessInformation
  nt!NtQuerySystemInformation

Both crashes show OUT_OF_SYNC_PROCESSOR – Windows is sending IPIs to vCPUs that are not responding, or querying Hyper-V enlightenment time sources that return inconsistent values across vCPUs. Hypervisor enlightenments were active in both dumps (Hypervisor.Flags.AnyHypervisorPresent: 1).


What did not help

• Adding args: hv-time to the VM config
• Changing CPU type to x86-64-v3
• Host reboot (on the same Kernel 7.0.2-7)
• SFC and DISM inside Windows

Current workaround
Pinning the host to Kernel 6.17.13-13-pve via proxmox-boot-tool kernel pin 6.17.13-13-pve and rebooting the host. The VM has been running stably since then. This strongly suggests a regression or incompatibility introduced in Kernel 7.x affecting Windows VMs under KVM with Hyper-V enlightenments.


Open questions
Has anyone else seen OUT_OF_SYNC_PROCESSOR BSODs in Windows VMs after upgrading to Kernel 7.x? Is this a known issue, and is there a proper fix rather than staying on 6.17 indefinitely?

 

[driley]

Hi,

I was testing Virtualization Based Security and Hyper-V recently and also experienced frequent crashes especially after migrations and general instability, in the form of freezes from 30-60 seconds and these two Blue Screens occurred regularly: "DPC Watchdog Violation" and "Clock Watchdog Timeout".

Before changing anything, it might be interesting to see what security properties Windows currently thinks are available. Could you run this in an elevated PowerShell prompt on the guest an post the output here?

(Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard).AvailableSecurityProperties

If it returns 8 in the list, it means hardware APIC virtualization is active, which is likely where the 7.x kernel is tripping up (Validate enabled VBS and memory integrity features).

If 8 is included in the list you could try to disable x2apic by creating a Custom CPU Model.
This can be configured under: Datacenter -> Custom CPU models -> Add
There you can just use "host" as your base model and search for the x2apic flag and disable it. After creating this you can assign it to the VM.

You can also edit the file directly. It is located in

/etc/pve/virtual-guest/cpu-models.conf

and has the following structure:

cpu-model: windows
        flags -x2apic
        reported-model host

Feel free to let me know if this helped.

 

[pmbaeum]

Thanks for the quick response!

The output of AvailableSecurityProperties is:
(Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard).AvailableSecurityProperties
1
2
4
5

No 8 in the list, so hardware APIC virtualization does not appear to be active according to Windows.

x2apic is not explicitly configured in the VM config, but since the CPU type is set to host, it likely gets inherited from the host CPU. I haven't created a custom CPU model to disable it yet.

Would you still recommend trying the x2apic approach despite the missing 8, or does the absence of it point to a different cause?

Note: During debugging I also switched the CPU type to x86-64-v3 – no change in behaviour. Dump 2 from the initial post was captured with that setting, still on Kernel 7.0.2-7.

 

[driley]

I would still give it a shot just to rule it out, but I get that you might not be able play around on the machine as you are depending on the Exchange Server being up. It might make sense to setup a small Windows Server VM and see if you can reproduce the issue on that as well.

Can you please show me the output of the following command on your PVE host:
qm showcmd <vmid> --pretty

This way we know exactly which flags are set.

Do you have Hyper-V or Virtualization Based Security enabled in Windows?
And what CPU are you on?

 

[pmbaeum]

Output of qm showcmd:

qm showcmd 131 --pretty
/usr/bin/kvm \
  -id 131 \
  -name EX-AHBSV03 \
  -no-shutdown \
  -chardev 'socket,id=qmp,path=/var/run/qemu-server/131.qmp,server=on,wait=off' \
  -mon 'chardev=qmp,mode=control' \
  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' \
  -mon 'chardev=qmp-event,mode=control' \
  -pidfile /var/run/qemu-server/131.pid \
  -daemonize \
  -smbios 'type=1,uuid=d4fac7e5-0327-4c1f-80ed-e7115a1d14a7' \
  -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' \
  -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd"},"node-name":"pflash0","read-only":true}' \
  -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/131/vm-131-disk-1.qcow2","node-name":"e1a199ceb9d952047290d2bd6ba60ab","read-only":false},"node-name":"f1a199ceb9d952047290d2bd6ba60ab","read-only":false},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"}' \
  -smp '12,sockets=1,cores=12,maxcpus=12' \
  -nodefaults \
  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
  -vnc 'unix:/var/run/qemu-server/131.vnc,password=on' \
  -cpu qemu64,+abm,+aes,+avx,+avx2,+bmi1,+bmi2,enforce,+f16c,+fma,+kvm_pv_eoi,+kvm_pv_unhalt,+movbe,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3,+xsave \
  -m 65536 \
  -object '{"id":"throttle-drive-ide0","limits":{},"qom-type":"throttle-group"}' \
  -object 'iothread,id=iothread-virtioscsi0' \
  -object '{"id":"throttle-drive-scsi0","limits":{},"qom-type":"throttle-group"}' \
  -object 'iothread,id=iothread-virtioscsi1' \
  -object '{"id":"throttle-drive-scsi1","limits":{},"qom-type":"throttle-group"}' \
  -object 'iothread,id=iothread-virtioscsi2' \
  -object '{"id":"throttle-drive-scsi2","limits":{},"qom-type":"throttle-group"}' \
  -object 'iothread,id=iothread-virtioscsi3' \
  -object '{"id":"throttle-drive-scsi3","limits":{},"qom-type":"throttle-group"}' \
  -object 'iothread,id=iothread-virtioscsi4' \
  -object '{"id":"throttle-drive-scsi4","limits":{},"qom-type":"throttle-group"}' \
  -global 'ICH9-LPC.disable_s3=1' \
  -global 'ICH9-LPC.disable_s4=1' \
  -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
  -device 'vmgenid,guid=f31c35f9-c471-46a2-8ed1-9f75dd3daf19' \
  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
  -chardev 'socket,id=tpmchar,path=/var/run/qemu-server/131.swtpm' \
  -tpmdev 'emulator,id=tpmdev,chardev=tpmchar' \
  -device 'tpm-tis,tpmdev=tpmdev' \
  -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
  -chardev 'socket,path=/var/run/qemu-server/131.qga,server=on,wait=off,id=qga0' \
  -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
  -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:xxxxxxx' \
  -blockdev '{"driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"driver":"file","filename":"/mnt/pve/WD8TB/template/iso/virtio-win-0.1.285.iso","node-name":"e93cbc830166a5959868e032a2c95ff","read-only":true},"node-name":"f93cbc830166a5959868e032a2c95ff","read-only":true},"node-name":"drive-ide0","read-only":true,"throttle-group":"throttle-drive-ide0"}' \
  -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=103' \
  -device 'ide-cd,bus=ide.1,unit=0,id=ide2,bootindex=101' \
  -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
  -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/var/lib/vz/images/131/vm-131-disk-0.qcow2","node-name":"e82f7735cb2f94b02b48608e469e645","read-only":false},"node-name":"f82f7735cb2f94b02b48608e469e645","read-only":false},"node-name":"drive-scsi0","read-only":false,"throttle-group":"throttle-drive-scsi0"}' \
  -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,device_id=drive-scsi0,rotation_rate=1,bootindex=100,write-cache=on' \
  -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2,iothread=iothread-virtioscsi1' \
  -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/zvol/ZFS8TB/vm-131-disk-0","node-name":"ec25dae585018d203a64378fbab8b94","read-only":false},"node-name":"fc25dae585018d203a64378fbab8b94","read-only":false},"node-name":"drive-scsi1","read-only":false,"throttle-group":"throttle-drive-scsi1"}' \
  -device 'scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,device_id=drive-scsi1,rotation_rate=1,write-cache=on' \
  -device 'virtio-scsi-pci,id=virtioscsi2,bus=pci.3,addr=0x3,iothread=iothread-virtioscsi2' \
  -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/zvol/ZFS8TB/vm-131-disk-1","node-name":"e398efccc8cb973ac2a56085a39796e","read-only":false},"node-name":"f398efccc8cb973ac2a56085a39796e","read-only":false},"node-name":"drive-scsi2","read-only":false,"throttle-group":"throttle-drive-scsi2"}' \
  -device 'scsi-hd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,device_id=drive-scsi2,rotation_rate=1,write-cache=on' \
  -device 'virtio-scsi-pci,id=virtioscsi3,bus=pci.3,addr=0x4,iothread=iothread-virtioscsi3' \
  -blockdev '{"detect-zeroes":"unmap","discard":"unmap","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/zvol/ZFS8TB/vm-131-disk-2","node-name":"ec1a2a6fbbc35c097bffee21794cc9e","read-only":false},"node-name":"fc1a2a6fbbc35c097bffee21794cc9e","read-only":false},"node-name":"drive-scsi3","read-only":false,"throttle-group":"throttle-drive-scsi3"}' \
  -device 'scsi-hd,bus=virtioscsi3.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi3,id=scsi3,device_id=drive-scsi3,rotation_rate=1,write-cache=on' \
  -device 'virtio-scsi-pci,id=virtioscsi4,bus=pci.3,addr=0x5,iothread=iothread-virtioscsi4' \
  -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"qcow2","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"file","filename":"/mnt/pve/WD8TB/images/131/vm-131-disk-0.qcow2","node-name":"ec9bb4e46979d07604b0274573f62a7","read-only":false},"node-name":"fc9bb4e46979d07604b0274573f62a7","read-only":false},"node-name":"drive-scsi4","read-only":false,"throttle-group":"throttle-drive-scsi4"}' \
  -device 'scsi-hd,bus=virtioscsi4.0,channel=0,scsi-id=0,lun=4,drive=drive-scsi4,id=scsi4,device_id=drive-scsi4,rotation_rate=1,write-cache=on' \
  -netdev 'type=tap,id=net0,ifname=tap131i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=BC:24:11:xx:xx:xx,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102,host_mtu=1500' \
  -machine 'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=q35+pve0' \
  -cpu host,-hv-time

A few things worth noting from the output:
- The args line -cpu host,-hv-time is active (added during debugging)
- No explicit Hyper-V enlightenment flags (hv_relaxed, hv_spinlocks etc.) are set
- hpet=off is set in the machine line

Host CPU is an Intel Xeon Silver 4314.

Regarding Hyper-V / VBS: I have not intentionally enabled either inside Windows. The dump analysis showed Hypervisor.Flags.AnyHypervisorPresent: 1, which I believe is just KVM advertising itself, not an actual Hyper-V/VBS setup.

Regarding the test VM: I am happy to test this in a maintenance window with a small Windows Server VM on Kernel 7.x to reproduce the issue in a controlled environment. That said, since I have already pinned the host to 6.17.13-13-pve and Exchange is running stably, I cannot do that right now during production hours.

Update: Colleague just told me we ordered another Server with Intel Gold 6138 - will try to use that one for debugging before it goes into production.
Will be ~1 week until it arrives.

 

[driley]

Thanks for the input.

I would recommend trying to disable:

args: -cpu host,level=30,-hv-time,-hv-stimer,-hv-stimer-direct,-x2apic

If it still crashes after that, you can try running a trace on your host to see exactly which KVM events are firing right before the Blue Screen. It isn't always straightforward since you can't manually force the BSOD, but it will capture the underlying hypervisor errors.

You will need to install the tool first: apt install trace-cmd

Then start the capture:

trace-cmd record -e kvm

Just a quick heads-up: a KVM trace on a busy host will generate a massive log very quickly. Be sure to keep an eye on your disk space and stop the trace (Ctrl+C) immediately after the VM crashes so it doesn't fill up your drive!

Feel free to keep me posted

 

[pmbaeum]

Hi,
Thanks for the suggestion. I applied the recommended args:

args: -cpu host,level=30,-hv-time,-hv-stimer,-hv-stimer-direct,-x2apic

No change in behaviour – the VM still crashes with DPC_WATCHDOG_VIOLATION on Kernel 7.0.6-2-pve.

I ran a trace-cmd record -e kvm during a crash and also checked journalctl -k -b -1. The kernel log from the previous boot shows two relevant findings:

1. Split lock traps starting ~2 minutes before the crash, escalating to a storm across all vCPUs:

Jun 09 20:27:02 pve9 kernel: x86/split lock detection: #AC: CPU 8/KVM/110494 took a split_lock trap at address: 0xfffff800448bd2ca
Jun 09 20:27:02 pve9 kernel: x86/split lock detection: #AC: CPU 5/KVM/110491 took a split_lock trap at address: 0xfffff800448bd2ca
Jun 09 20:27:02 pve9 kernel: x86/split lock detection: #AC: CPU 6/KVM/110492 took a split_lock trap at address: 0xfffff800448bd2ca
[... multiple CPUs, same address 0xfffff800448bd2ca ...]

2. Kernel WARNING in vmx_check_nested_events at crash time:

Jun 09 20:32:07 pve9 kernel: WARNING: arch/x86/kvm/vmx/nested.c:4462 at vmx_check_nested_events+0x910/0x920 [kvm_intel]
Jun 09 20:32:07 pve9 kernel: CPU: 48 UID: 0 PID: 110489 Comm: CPU 3/KVM Tainted: P O 7.0.6-2-pve #1 PREEMPT(lazy)
Jun 09 20:32:07 pve9 kernel: Call Trace:<br></span><span>Jun 09 20:32:07 pve9 kernel:  kvm_check_nested_events+0x22/0x50 [kvm]
Jun 09 20:32:07 pve9 kernel:  kvm_check_and_inject_events+0x26c/0x550 [kvm]<br></span><span>Jun 09 20:32:07 pve9 kernel:  kvm_arch_vcpu_ioctl_run+0x489/0x18c0 [kvm]

This points to a bug in the nested VMX event injection path in kvm_intel on Kernel 7.0.x.
Disabling the Hyper-V enlightenment flags via args has no effect – the WARNING still occurs.


Update on the 6.17 workaround:
Pinning to 6.17.13-13-pve is not fully stable either – the VM also crashed once with CLOCK_WATCHDOG_TIMEOUT (0x101) on that kernel. The frequency is significantly lower compared to 7.0.x, but the underlying issue appears to affect both kernel generations. This suggests the root cause may be deeper in the KVM/nested VMX interaction with Windows Server 2025 Hyper-V enlightenments rather than a clean 7.0-specific regression.

I have a full trace-cmd capture (1.8 GB, -e kvm, covering the crash on 7.0.6-2-pve) available if that would be useful for analysis. Could upload it on my nextcloud, but would prefer not to send it publicly?