VM freeze on kernel 7.0.x (7.0.6 → 7.0.14) — vCPUs stuck in KVM_RUN, recovers instantly on ptrace attach — stable on 7.0.2

[topkung1]

Hi all,

After the kernel moved from the 7.0.2 base to 7.0.6-2-pve, my Windows VMs
intermittently freeze and only a hard reset recovers them. The 7.0.2 base
(the PVE 9.2 default) has been stable. I am currently pinned back to
7.0.2-7-pve and observing.

Symptoms
- A VM becomes completely unresponsive: RDP disconnects, the noVNC console
shows only a black screen, and the qemu-guest-agent guest-ping times out.
- On the host, the kvm process for the affected VM is pegged at ~1100% CPU
(compute-bound, not I/O wait). Host RAM is healthy (plenty free, swap unused).
- dmesg is full of "x86/split lock detection: #AC ... took a split_lock trap"
messages — some at user-space addresses (0x7ff7...), some in the guest
kernel (0xfffff80...).
- It has happened on two VMs with very different configs (one q35 + OVMF,
the other i440fx + SeaBIOS), so it doesn't look VM-config specific. One of
them was only running a web browser when it froze.
- Only `qm reset` / `qm stop` recovers it.

Host
- CPU: Intel Core i9-14900K (single socket)
- RAM: 64 GB, non-ECC
- Motherboard: ASRock Rack Z690D4U, BIOS 21.12 (2025-11-28)
- Storage: NVMe + SATA SSD
- intel-microcode: 3.20251111.1~deb13u1 (recent — so this is not stale microcode)
- PVE: proxmox-ve 9.2.0, pve-manager 9.2.3, pve-qemu-kvm 11.0.0-4
- Affected kernel: 7.0.6-2-pve | Stable kernel: 7.0.2-7-pve (also 7.0.2-6)

What I've tried
- `sysctl kernel.split_lock_mitigate=0`: does NOT rescue a VM that is already
hung; testing whether it prevents recurrence.
- Pinned 7.0.2-7-pve via `proxmox-boot-tool kernel pin` and rebooted; so far
observing for stability.

This looks very similar to another report here:

G

Thread 'Proxmox VE compatibility issue with Intel I9-14900K?'

Jun 14, 2026

I just reinstalled Proxmox VE (9.2-1) on my Homeserver (Asus Rog Strix Z690-E Mainboard with Intel Core I9-14900K) because the server freezes while booting at different points. After fresh installation with "proxmox-kernel-7.0.2-6-pve-signed" reboot works. After updating to "proxmox-kernel-7.0.6-2-pve-signed", the server freezes again during boot process. Is there a compatibility issue with mainboard or cpu? I tried to find logs with specific errors, but nothing usefull. Memtest and cpu tests without errors, latest bios version installed. I now changed grub to boot "7.0.2-6". Thanks for help.

• Gating7897
• 7.0.6-2 boot freeze i9-14900k kernel kernel 7.0.2-6-pve z690-e
• Replies: 4
• Forum: Proxmox VE: Installation and configuration

Questions
- Is this a known regression in the 7.0.6 base on Intel Raptor Lake (13th/14th
gen) hosts?
- Did split_lock_detect / bus-lock handling change between the 7.0.2 and 7.0.6
bases?
- Any recommended fix beyond pinning 7.0.2 / split_lock_mitigate=0, or is a fix
expected in a newer 7.0.x kernel?

Full `pveversion -v`, dmesg, and `qm config` available on request.

Thanks!

 

[Maximiliano]

Regarding split lock detection, this is done on the kernel side. Please take a look at [1] for more information. Each time a split_lock is detected the CPU the kernel freezes the CPU for 10ms which can tank performance if it happens too often, unfortunately, the impact of a split lock cannot be determined by looking at the journal as it is only logged once, see [1] on how to check how often they are detected.


[1] https://pve.proxmox.com/wiki/Split_lock_detection

 

[topkung1]

Update — I caught the hang live this time, and the evidence points at a missed wakeup in KVM.

It's the kernel, not QEMU. Same pve-qemu-kvm 10.1.2-7 throughout:

• kernel 7.0.2-6 → 9.5 days, no hang
• kernel 7.0.14-4 → hung within 9 hours

And for context, this box ran 6.17.x for 5 months (uptimes up to 20 days) with zero hangs. Every 7.0.x kernel I've tried hangs, and it's getting worse: 7.0.6 took ~7 days to first hang, 7.0.14 hangs within hours.

Only the kernel changed.

What I saw during the hang:

• Host CPU high (~50% plateau, load avg ~18) — vCPUs were spinning
• QEMU main loop perfectly healthy (ppoll, 3s idle timeout, no locks held) — no QEMU-side deadlock
• Guest event log: last entry 23:38:49 (a routine cert update), then complete silence until 00:00:25. 21 minutes, not a single entry. Windows wasn't slow — it wasn't executing.
• All 24 vCPU threads stuck in ioctl(KVM_RUN):

#0 ioctl()
#1 kvm_vcpu_ioctl (type=44672)   <- 0xAE80 = KVM_RUN
#2 kvm_cpu_exec
#3 kvm_vcpu_thread_fn

The part that gave it away: I attached strace/gdb to grab a backtrace and the VM instantly recovered. Host CPU dropped straight back to normal. qemu-ga answered the ping it had been sitting on, services resumed, SQL Server was still on the same PID it had held for hours. Nothing in the guest crashed — it just woke up.

ptrace SIGSTOPs every thread, which kicks the vCPUs out of KVM_RUN with EINTR. They re-enter, KVM re-evaluates pending interrupts, and the guest wakes.

That matches the classic posted-interrupt failure almost word for word: a vCPU HLTs, its wakeup never gets delivered, the other vCPUs busy-spin waiting on it (hence the high CPU), and a forced VM-exit unblocks the whole thing. Not a deadlock, not the guest's fault.

Also worth noting: it hung while the VM was essentially idle. Low load means more HLTs, which would mean more chances to lose a wakeup.

Has anyone else hit this on 7.0.x? And for anyone on the KVM side — did anything change in the interrupt delivery / posted-interrupt / halt path between 7.0.2 and 7.0.6 that could drop a wakeup? Happy to run any test or patch.

Host: i9-14900K (8P+16E), ASRock Rack Z690D4U, 64GB non-ECC. No MCE/WHEA anywhere. The host itself never hangs — only the VM. Guest is Windows (q35/OVMF/TPM, cpu=host) running a game server plus SQL Server 2016.

proxmox-ve: 9.2.0 (running kernel: 7.0.14-4-pve)
pve-manager: 9.2.4 (running version: 9.2.4/5e5ae681198514d4)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-4-pve-signed: 7.0.14-4
proxmox-kernel-7.0: 7.0.14-4
proxmox-kernel-7.0.14-3-pve-signed: 7.0.14-3
proxmox-kernel-7.0.12-1-pve-signed: 7.0.12-1
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-7.0.2-7-pve-signed: 7.0.2-7
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
proxmox-kernel-6.17: 6.17.13-15
proxmox-kernel-6.17.13-15-pve-signed: 6.17.13-15
proxmox-kernel-6.17.13-14-pve-signed: 6.17.13-14
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.16
libpve-guest-common-perl: 6.0.4
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.6
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.2-1
proxmox-backup-file-restore: 4.2.2-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.6
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.3
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-4
pve-ha-manager: 5.2.4
pve-i18n: 3.9.0
pve-qemu-kvm: 10.1.2-7
pve-xtermjs: 6.0.0-2
qemu-server: 9.1.18
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.3-pve1

agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 24
cpu: host
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,ms-cert=2023k,pre-enrolled-keys=1,size=4M
machine: pc-q35-10.1
memory: 20480
meta: creation-qemu=9.0.2,ctime=1728243092
name: win-guest
net0: virtio=BC:24:11:78:F2:FC,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
protection: 1
scsi0: local-lvm:vm-101-disk-1,cache=writeback,discard=on,iothread=1,size=390G
scsihw: virtio-scsi-single
sockets: 1
tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0

cpus: 24
maxmem: 21474836480
mem: 21710696448
pid: 48353
pressurecpufull: 0
pressurecpusome: 0
pressureiofull: 0
pressureiosome: 0
pressurememoryfull: 0
pressurememorysome: 0
qmpstatus: running
running-machine: pc-q35-10.1+pve0
running-qemu: 10.1.2
status: running
uptime: 23372
vmid: 101

(scsi0 blockstat)
failed_flush_operations: 0
failed_rd_operations: 0
failed_wr_operations: 0
invalid_flush_operations: 0
invalid_rd_operations: 0
invalid_wr_operations: 0
flush_operations: 13787
rd_operations: 978867
wr_operations: 800238

 

[Maximiliano]

Hello,

We aware of a issue that results in Windows 11/2022/2025 VMs using the `host` CPU type freezing for some time (10-30 minutes) and 100% CPU usage can be seen on the VM's summary. This happens on kernels 7.0.0 or newer (tested up to 7.0.14) when using certain Intel CPUs models. We are currently investigating the issue.

At the moment the recommendation is to either stick with kernel 6.17 or to use a CPU type different from host.

 

[topkung1]

Thanks a lot — good to know it's a known issue and being looked at.

I've pinned kernel 6.17.13-15-pve and it's been up 1 day 10 hours so far with no freeze. For comparison, 7.0.14-4 froze within 9 hours on this box, so that's already a good sign, but I'll keep watching.

If it happens again I'll report back with strace + gdb backtraces from the live hang.

Thanks again!

 

[Maximiliano]

I edited my previous message to nail down a bit further the affected setups.

 

[jtru]

@Maximiliano thanks for this important clarification - we have a situation where many of our Windows Server 2022 VMs *permanently* (until a hard reboot/SIGTERM to the qemu process, and ptrace(2) does not transiently fix the problem) freeze/hang after variable amounts of uptime (between hours and weeks, actually, with no clear pattern emerging) on AMD EPYC 9175F running 7.0.2-6-pve (2 clusters with 5 nodes each).

I take it you do not have that kind of problem on your radar as of today?

 

[Maximiliano]

It looks like something different then.

Please take a look at [1] for more details. It can happen that if there are multiple VSS writers inside of a window guest, when a guest-fsfreeze-freeze command is issued to the QEMU guest agent, the Windows OS will not react appropriately and freeze altogether (note that these are two different kind of freezes, the former is a controlled freeze for filesystem IO only while the later is a complete unintended freeze). Please see [2] for more details on the guest-fsfreeze commands.

As a workaround, [1] Suggests to se the VSS mode to COPY (option 5). If that does not help you can disable issuing guest-fsfreeze commands altogether as described at [2] via the freeze-fs option.

[1] https://pve.proxmox.com/wiki/VM_Backup_Consistency
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_qga_fsfreeze

 

[robertlukan]

I can also confirm Windows VMs freezing if they have only host feature enabled. It is PVE 9.2.4 update from enterprise repo last week.

64 x Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (2 Sockets)
Linux 7.0.14-4-pve (2026-07-07T07:27Z

If a VM has the following args it is fine:

args: -cpu host,+vmx,+kvm_pv_eoi,+kvm_pv_unhalt,hv-relaxed,hv-vapic,hv-time,hv-synic,hv-stimer,hv-vpindex,hv-ipi,hv-runtime,hv-reset,hv-crash,hv-spinlocks=0xfff,hv-tlbflush,hv-tlbflush-ext,hv-tlbflush-direct,hv-stimer-direct,hv-apicv,hv-avic,hv-evmcs,hv-xmm-input,hv-emsr-bitmap,host-cache-info=on,l3-cache=on,+invtsc,tsc-frequency=2900000000

I will create a blog post why I need those flags, please do not use them blindly.

So this will be a hint now for Proxmox staff why WIndows 11 with just host flag is crashing and with those flags is not.

 

[j.theisen]

Hi @robertlukan

args: -cpu host,+vmx,+kvm_pv_eoi,+kvm_pv_unhalt,hv-relaxed,hv-vapic,hv-time,hv-synic,hv-stimer,hv-vpindex,hv-ipi,hv-runtime,hv-reset,hv-crash,hv-spinlocks=0xfff,hv-tlbflush,hv-tlbflush-ext,hv-tlbflush-direct,hv-stimer-direct,hv-apicv,hv-avic,hv-evmcs,hv-xmm-input,hv-emsr-bitmap,host-cache-info=on,l3-cache=on,+invtsc,tsc-frequency=2900000000

I will create a blog post why I need those flags, please do not use them blindly.

Thank you for your testing! I have also looked into this issue a bit.
I know you said you'll create a blog post and please link it as soon as its available.
Nevertheless i'd like to ask about the specific flag hv-evmcs. Why is this needed in your case and what does it change for you?

Also:

It is PVE 9.2.4 update from enterprise repo last week.

pve-manager 9.2.4 hasn't landed in enterprise repo yet, so in case this is not a typo, you might still have the no-subscription or testing repos configured on this machine.

Best regards
Jonas

 

[robertlukan]

Yeah 9.2.4 is a typo, I am using that in my lab. It is 9.2.3, sorry my mistake.

I am in the process of migrating about 100+ Windows 11 Virtual desktops from Vmware to Proxmox. I wanted to have VBS, WSL 2 running. I noticed if I use hv-passthrough flag everything works fine, BUT migration does not work. So I captured all the flags that were passed with hv-passthrough and added invtsc and tsc-frequency to regain the migration.

So it really depends what your cpu can offer I guess, so please do not use those flags blindly.

This specific setup is only needed if you dont have newer CPUs, in my lab setup with Intel and AMD desktop CPUs it is not needed to define the invtsc and tsc-frequency.

I am still creating a blog with all those details, maybe it will be useful for someone. I am trying to include all those details that I am still learning.

I hope this explains(somehow) my comment.