Guest VMs freeze 10-250 s (missed vCPU wakeup) on PVE 9.x kernels 6.17.x and 7.0.x
on HP ProLiant DL380 Gen9 (Haswell E5-2650 v3) hosts - 6.14.11-x is clean
=====================================================================================
Summary
-------
On a 3-node PVE 9.2 cluster of identical HP DL380 Gen9 hosts (2x Xeon E5-2650 v3,
Haswell, 251 GiB, BIOS P89 11/23/2021, microcode 0x49), guests running on kernel
6.17.13-x or 7.0.14-x periodically stop executing for 10 s to several minutes.
The host logs nothing and is nearly idle. The same guests on 6.14.11-4 / 6.14.11-9
on the same hardware are clean, and identical guest images on our other cluster
(Broadwell / Skylake, PVE 8, kernel 6.8.12) have never shown it.
We spent ~10 weeks assuming this was load-related (a PBS verify job) and it is not.
The cleanest reproduction is a 4-vCPU HAProxy VM that is the only guest on a
40-thread host with the host at 98 % idle.
Environment
-----------
- Hosts: 3x HP ProLiant DL380 Gen9, 2x Intel Xeon E5-2650 v3 @ 2.30GHz (Haswell-EP),
20C/40T, 16x 16 GB DDR4-2133, BIOS P89 11/23/2021, microcode 0x49.
KVM: kvm_intel enable_apicv=Y, ept=Y, halt_poll_ns=200000 (defaults).
- PVE: proxmox-ve 9.2.0, pve-manager 9.2.11, pve-qemu-kvm 11.0.3-3, qemu-server 9.2.7
- Shared storage: HP MSA 2040 SAS (LVM over multipath), local ZFS mirrors, NFS.
- Kernels tested on this hardware:
6.14.11-4-pve clean (71 days, HAProxy VM: 0 stalls)
6.14.11-9-pve now pinned everywhere (see below)
6.17.13-1/-2 guest soft lockups in tcp_delack_timer / __pv_queued_spin_lock_slowpath
6.17.13-13 fewer, but still ongoing (smpp gateway VM: 7 lockups in 3 days last week)
7.0.14-14-pve PBS VM: 70 lockups during a single Saturday verify job, one stall 245 s
7.0.14-15-pve HAProxy VM: 3 whole-VM freezes of 11 / 24 / 59 s in the first 24 h,
~19 health-check timeout bursts in 2.3 days, then HAProxy's own
watchdog aborted the process
- Guests affected: Debian 12 PBS (kernel 6.1, 8 vCPU cpu=host), Ubuntu 22.04
(5.15, 4 vCPU cpu=host, q35, virtio-scsi-single, virtio-net), Ubuntu nginx / smpp
VMs. Windows 11 was also on the host but is unmeasurable.
Symptoms
--------
1. Under load (PBS verify), classic guest soft lockups:
watchdog: BUG: soft lockup - CPU#n stuck for 20s..245s!
smp_call_function_many_cond+0x2e7 (kvm_flush_tlb_multi -> TLB shootdown IPI
to a vCPU that never answers)
__pv_queued_spin_lock_slowpath (lock holder vCPU is halted)
swapper lockups (timer irq never delivered to an idle vCPU)
Stuck tasks were named [verify chunk ...], [tokio-runtime-w], [kworker], [swapper].
systemd inside the guest killed journald/logind/udevd on its own watchdog during
the 245 s stall. Backups and verifies still completed OK.
2. Under NO load (HAProxy VM), the entire VM stops. There are NO soft-lockup lines
because every vCPU is halted so the guest watchdog itself never runs. What you
see instead:
kernel: clocksource: Long readout interval, skipping watchdog check:
cs_nsec: 58990674636 wd_nsec: 58990674976 (= 59 s gap)
and in HAProxy, on resume, every backend health check expiring at the same
microsecond:
Server mysql_write/mysql3 is DOWN, reason: Layer4 timeout, check duration: 2001ms
Server mysql_read/mysql2 is DOWN, reason: Layer4 timeout, check duration: 2000ms
... (11 servers, identical timestamp) ...
[ALERT] proxy 'mysql_combined' has no server available!
followed within 1 s by all of them coming back UP with 27-57 ms checks.
Eventually HAProxy's internal watchdog fired:
Thread 1 is about to kill the process.
stuck=1 ... cpu_ns: poll=497151958526 now=499301360360 diff=2149401834
fct=process_chk ... wdt_handler+0x1ab
Current worker #1 (774) exited with code 134 (Aborted)
i.e. a thread that had not progressed for 2.1 s of wall time while in a trivial
health-check function. systemd restarted haproxy; all client connections dropped.
Things ruled out on the host (during the PBS stalls)
----------------------------------------------------
- Host load 3.5 / 40 threads, CPU 8 %, 98 % idle.
- No host-side kernel messages, no soft lockups, no MCE, no storage / multipath errors.
- No memory pressure: allocstall, pgscan_direct, compact_stall all 0, PSI memory 0,
KSM off, THP madvise, ZFS ARC 9/16 G.
- Guest steal time since boot: 37 s total over weeks (PBS), 15 s (HAProxy).
So vCPUs are not being descheduled while runnable - they are HALTED (idle,
cpuidle driver "none" in guest, plain HLT) and the host does not wake them.
- Not backup / snapshot / migration timed; not tied to any host cron.
- Same guest image (a second HAProxy VM) on a Skylake host with 6.8.12 = 0 events.
Timeline of evidence
--------------------
- 2026-06-24: pve6/pve7 on 6.17.13-1/-2. PBS VM 598 lockups, puppeteer VM 219,
HAProxy 15. Migrated HAProxy to a third identical host pinned to 6.14.11-4:
0 stalls for the next 71 days (the control). Pinned pve6/7 to 6.17.13-13: the
tcp_delack network-stall variant went away, the TLB-IPI variant did not.
- 2026-09-03: pve6 -> 7.0.14-14 as a test. Saturday verify: 70 lockups, one of 245 s.
- 2026-09-04: pve8 (the clean control host) -> 7.0.14-15. HAProxy started freezing
within 5 minutes of the guest booting (first 11 s clocksource gap at +5 min).
- 2026-09-06: HAProxy watchdog SIGABRT. Pinned pve8 and pve6 back to 6.14.11-9.
pve8 rebooted onto it; monitoring.
Why we think it is the kernel, not the hardware alone
-----------------------------------------------------
Same hardware, same BIOS/microcode, same guests, same QEMU 11.0.3: 6.14.11-4 clean
for 71 days, 7.0.14-15 stalls within minutes. Only the host kernel changed. That
said, we cannot exclude a Haswell-specific interaction (APICv / posted-interrupt
wakeup path); our Broadwell and Skylake hosts have never run a kernel newer than
6.8.12, so "newer silicon is fine" is not proven either.
Workaround
----------
proxmox-boot-tool kernel pin 6.14.11-9-pve && proxmox-boot-tool refresh && reboot
6.14.11-9 differs from the proven 6.14.11-4 only by the Ubuntu 6.14.0-37 rebase, an
ice NIC leak fix and the 2026 security cherry-picks; nothing KVM related.
Questions for the Proxmox team / others
---------------------------------------
- Is anyone else seeing halted vCPUs not being woken on Haswell-EP with 6.17 / 7.0?
- Is there a known fix in a later 7.0.x or 6.17.13-2x? We did not find one in the
changelogs.
- Would kvm_intel enable_apicv=0 or idle=poll in the guest be useful data points?
(We have not tried them yet; happy to.)
Detection tips for others
-------------------------
- grep the guest for "Long readout interval" - that catches the silent, no-load
whole-VM stalls that never produce a soft-lockup line.
- In HAProxy, many backends going DOWN with "Layer4 timeout" at the identical
timestamp and coming back within a second is the VM having been frozen, not the
backends.
- A 1-minute Zabbix TCP check only catches stalls that straddle a poll; we saw 5
alerts for ~19 real events.
on HP ProLiant DL380 Gen9 (Haswell E5-2650 v3) hosts - 6.14.11-x is clean
=====================================================================================
Summary
-------
On a 3-node PVE 9.2 cluster of identical HP DL380 Gen9 hosts (2x Xeon E5-2650 v3,
Haswell, 251 GiB, BIOS P89 11/23/2021, microcode 0x49), guests running on kernel
6.17.13-x or 7.0.14-x periodically stop executing for 10 s to several minutes.
The host logs nothing and is nearly idle. The same guests on 6.14.11-4 / 6.14.11-9
on the same hardware are clean, and identical guest images on our other cluster
(Broadwell / Skylake, PVE 8, kernel 6.8.12) have never shown it.
We spent ~10 weeks assuming this was load-related (a PBS verify job) and it is not.
The cleanest reproduction is a 4-vCPU HAProxy VM that is the only guest on a
40-thread host with the host at 98 % idle.
Environment
-----------
- Hosts: 3x HP ProLiant DL380 Gen9, 2x Intel Xeon E5-2650 v3 @ 2.30GHz (Haswell-EP),
20C/40T, 16x 16 GB DDR4-2133, BIOS P89 11/23/2021, microcode 0x49.
KVM: kvm_intel enable_apicv=Y, ept=Y, halt_poll_ns=200000 (defaults).
- PVE: proxmox-ve 9.2.0, pve-manager 9.2.11, pve-qemu-kvm 11.0.3-3, qemu-server 9.2.7
- Shared storage: HP MSA 2040 SAS (LVM over multipath), local ZFS mirrors, NFS.
- Kernels tested on this hardware:
6.14.11-4-pve clean (71 days, HAProxy VM: 0 stalls)
6.14.11-9-pve now pinned everywhere (see below)
6.17.13-1/-2 guest soft lockups in tcp_delack_timer / __pv_queued_spin_lock_slowpath
6.17.13-13 fewer, but still ongoing (smpp gateway VM: 7 lockups in 3 days last week)
7.0.14-14-pve PBS VM: 70 lockups during a single Saturday verify job, one stall 245 s
7.0.14-15-pve HAProxy VM: 3 whole-VM freezes of 11 / 24 / 59 s in the first 24 h,
~19 health-check timeout bursts in 2.3 days, then HAProxy's own
watchdog aborted the process
- Guests affected: Debian 12 PBS (kernel 6.1, 8 vCPU cpu=host), Ubuntu 22.04
(5.15, 4 vCPU cpu=host, q35, virtio-scsi-single, virtio-net), Ubuntu nginx / smpp
VMs. Windows 11 was also on the host but is unmeasurable.
Symptoms
--------
1. Under load (PBS verify), classic guest soft lockups:
watchdog: BUG: soft lockup - CPU#n stuck for 20s..245s!
smp_call_function_many_cond+0x2e7 (kvm_flush_tlb_multi -> TLB shootdown IPI
to a vCPU that never answers)
__pv_queued_spin_lock_slowpath (lock holder vCPU is halted)
swapper lockups (timer irq never delivered to an idle vCPU)
Stuck tasks were named [verify chunk ...], [tokio-runtime-w], [kworker], [swapper].
systemd inside the guest killed journald/logind/udevd on its own watchdog during
the 245 s stall. Backups and verifies still completed OK.
2. Under NO load (HAProxy VM), the entire VM stops. There are NO soft-lockup lines
because every vCPU is halted so the guest watchdog itself never runs. What you
see instead:
kernel: clocksource: Long readout interval, skipping watchdog check:
cs_nsec: 58990674636 wd_nsec: 58990674976 (= 59 s gap)
and in HAProxy, on resume, every backend health check expiring at the same
microsecond:
Server mysql_write/mysql3 is DOWN, reason: Layer4 timeout, check duration: 2001ms
Server mysql_read/mysql2 is DOWN, reason: Layer4 timeout, check duration: 2000ms
... (11 servers, identical timestamp) ...
[ALERT] proxy 'mysql_combined' has no server available!
followed within 1 s by all of them coming back UP with 27-57 ms checks.
Eventually HAProxy's internal watchdog fired:
Thread 1 is about to kill the process.
stuck=1 ... cpu_ns: poll=497151958526 now=499301360360 diff=2149401834
fct=process_chk ... wdt_handler+0x1ab
Current worker #1 (774) exited with code 134 (Aborted)
i.e. a thread that had not progressed for 2.1 s of wall time while in a trivial
health-check function. systemd restarted haproxy; all client connections dropped.
Things ruled out on the host (during the PBS stalls)
----------------------------------------------------
- Host load 3.5 / 40 threads, CPU 8 %, 98 % idle.
- No host-side kernel messages, no soft lockups, no MCE, no storage / multipath errors.
- No memory pressure: allocstall, pgscan_direct, compact_stall all 0, PSI memory 0,
KSM off, THP madvise, ZFS ARC 9/16 G.
- Guest steal time since boot: 37 s total over weeks (PBS), 15 s (HAProxy).
So vCPUs are not being descheduled while runnable - they are HALTED (idle,
cpuidle driver "none" in guest, plain HLT) and the host does not wake them.
- Not backup / snapshot / migration timed; not tied to any host cron.
- Same guest image (a second HAProxy VM) on a Skylake host with 6.8.12 = 0 events.
Timeline of evidence
--------------------
- 2026-06-24: pve6/pve7 on 6.17.13-1/-2. PBS VM 598 lockups, puppeteer VM 219,
HAProxy 15. Migrated HAProxy to a third identical host pinned to 6.14.11-4:
0 stalls for the next 71 days (the control). Pinned pve6/7 to 6.17.13-13: the
tcp_delack network-stall variant went away, the TLB-IPI variant did not.
- 2026-09-03: pve6 -> 7.0.14-14 as a test. Saturday verify: 70 lockups, one of 245 s.
- 2026-09-04: pve8 (the clean control host) -> 7.0.14-15. HAProxy started freezing
within 5 minutes of the guest booting (first 11 s clocksource gap at +5 min).
- 2026-09-06: HAProxy watchdog SIGABRT. Pinned pve8 and pve6 back to 6.14.11-9.
pve8 rebooted onto it; monitoring.
Why we think it is the kernel, not the hardware alone
-----------------------------------------------------
Same hardware, same BIOS/microcode, same guests, same QEMU 11.0.3: 6.14.11-4 clean
for 71 days, 7.0.14-15 stalls within minutes. Only the host kernel changed. That
said, we cannot exclude a Haswell-specific interaction (APICv / posted-interrupt
wakeup path); our Broadwell and Skylake hosts have never run a kernel newer than
6.8.12, so "newer silicon is fine" is not proven either.
Workaround
----------
proxmox-boot-tool kernel pin 6.14.11-9-pve && proxmox-boot-tool refresh && reboot
6.14.11-9 differs from the proven 6.14.11-4 only by the Ubuntu 6.14.0-37 rebase, an
ice NIC leak fix and the 2026 security cherry-picks; nothing KVM related.
Questions for the Proxmox team / others
---------------------------------------
- Is anyone else seeing halted vCPUs not being woken on Haswell-EP with 6.17 / 7.0?
- Is there a known fix in a later 7.0.x or 6.17.13-2x? We did not find one in the
changelogs.
- Would kvm_intel enable_apicv=0 or idle=poll in the guest be useful data points?
(We have not tried them yet; happy to.)
Detection tips for others
-------------------------
- grep the guest for "Long readout interval" - that catches the silent, no-load
whole-VM stalls that never produce a soft-lockup line.
- In HAProxy, many backends going DOWN with "Layer4 timeout" at the identical
timestamp and coming back within a second is the VM having been frozen, not the
backends.
- A 1-minute Zabbix TCP check only catches stalls that straddle a poll; we saw 5
alerts for ~19 real events.