Hi all,
I've been chasing an intermittent hang on a Windows Server 2022 guest and I'm out of ideas on the host side. Hoping someone recognizes the pattern.
Symptom
The guest becomes completely unresponsive, either a black console or frozen on the Windows logon screen in noVNC. Network dies, RDP dies, guest agent stops responding. From the Proxmox side the VM still shows as "running", no reset, no qmstop, nothing in the logs. I have to force-stop and start it to recover. Timing is unpredictable: sometimes stable 1 to 2 days, sometimes it hangs within minutes of boot and then repeatedly.
I originally assumed a specific LOB app in that guest was to blame, but I spun up a brand new clean Server 2022 VM with nothing installed and it hangs the same way, so I'm fairly sure this is host/config level, not the workload.
Host hardware (Dell PowerEdge R760xs)
Software stack
Guest config (clean repro VM is essentially identical)
What I've checked on the host, all clean
ECC / memory (EDAC), 0 errors on all controllers:
No machine-check / hardware errors:
NVMe SMART clean (0 media errors, 0 error-log entries, temps normal). ZFS pools ONLINE, 0 read/write/checksum errors. iDRAC hardware inventory shows all CPUs, DIMMs, and the PERC/battery healthy.
The key part: when the guest hangs, the host kernel log shows nothing at all. No OOM, no split-lock, no KVM error, no segfault, no soft lockup. The only VM-related lines are the tap interface going up/down from my own manual restarts:
journalctl -u qemu-server@104 -> no entries. No signal/kill logged against the qemu process other than my deliberate qmstop.
Guest side
Windows only logs the aftermath, Event 41 (Kernel-Power, "rebooted without cleanly shutting down") and 6008 (unexpected shutdown), which I believe are just the result of my own forced stop, not a spontaneous bugcheck. No MEMORY.DMP, no minidump, no BugCheck (1001) event, consistent with a hang rather than a Windows crash. I corrected a bad CrashDumpEnabled value (was 7, now 3/minidump) so a real bugcheck would be captured, but it just hangs silently, which wouldn't dump anyway.
Note: kernel cmdline has split_lock_detect=off, but the kernel reports Unknown kernel command line parameters "split_lock_detect=off", will be passed to user space, so it isn't taking effect that way. kernel.split_lock_mitigate is set to 0 via sysctl. Either way, no split-lock traps appear when it hangs.
Microcode (checked across all channels, already current, ruling it out)
The kernel logs "x86/CPU: Running old microcode" and the bugs line shows old_microcode + vmscape, so I initially suspected a microcode errata. But the CPU is already on the newest microcode that exists for this signature: running rev 0x2b000670, and Intel's latest release (microcode-20260512) still lists 0x2b000670 as the current revision for 06-8f-08/87 (platform ID 0), i.e. nothing newer exists for this stepping. I understand some MCUs ship only via OEM BIOS and never in the OS package, so I also confirmed the Dell BIOS (2.10.1, latest) currently delivers the same 0x2b000670. Both the OS package and BIOS are maxed out, there's no newer microcode to apply from either channel today. Flagging it in case the "old microcode" warning itself is meaningful, but I don't believe microcode is the cause.
Important cross-check
I run the same Proxmox + Windows Server setup on older Dell PowerEdge R640 hosts (Cascade Lake Xeons) with zero issues on any Windows version. The problem only appears on this newer R760xs (Sapphire Rapids, Xeon Silver 4509Y). The guest config is identical across both, so this points at a platform/kernel interaction specific to the newer CPU rather than the VM config. I can't pin the exact date, but my recollection is that the hangs started roughly a month ago, around the time I moved to the 7.x kernel branch.
Config changes implemented so far (currently testing for stability)
Interim workaround (symptom only, not a fix)
Because the hangs mostly happen during business hours, I've deployed the community "Monitor-All" script (ping-instances), scoped via tag to only the affected VM, so it auto-resets that one guest when its agent stops responding.
Something I'm considering but haven't done yet
Rolling the host back to a 6.x kernel to test whether this is a 7.x-kernel regression on this platform. Before I take the host down for that, I'd appreciate opinions: is a 6.x rollback worth trying here, or is there a better/faster way to isolate whether this is kernel vs. cpu-model vs. QEMU machine version? I'd rather not change several variables blindly.
Questions
Happy to provide any additional logs or config. Thanks in advance.
I've been chasing an intermittent hang on a Windows Server 2022 guest and I'm out of ideas on the host side. Hoping someone recognizes the pattern.
Symptom
The guest becomes completely unresponsive, either a black console or frozen on the Windows logon screen in noVNC. Network dies, RDP dies, guest agent stops responding. From the Proxmox side the VM still shows as "running", no reset, no qmstop, nothing in the logs. I have to force-stop and start it to recover. Timing is unpredictable: sometimes stable 1 to 2 days, sometimes it hangs within minutes of boot and then repeatedly.
I originally assumed a specific LOB app in that guest was to blame, but I spun up a brand new clean Server 2022 VM with nothing installed and it hangs the same way, so I'm fairly sure this is host/config level, not the workload.
Host hardware (Dell PowerEdge R760xs)
- 2x Intel Xeon Silver 4509Y (CPUID 06-8f-08, Sapphire Rapids-SP / Xeon Scalable Gen4, model 143 stepping 8), 8c each, 16 physical cores total
- 256 GB DDR5 (8x 32GB dual-rank, 4400 MT/s, Multi-bit ECC), balanced 4+4 across both sockets. All DIMMs healthy, 0 ECC errors.
- PERC H755 (FW 52.30.0-6753), battery Ready
- Boot: 2x SATA SSD (446 GB) as RAID-1 on the PERC (R1-ProxmoxOS)
- 4x SAS SSD (894 GB) exposed as individual RAID-0 "NonRAID" volumes on the PERC, used for a ZFS pool (R10-SSD). I'm aware ZFS-on-HW-RAID is not ideal; noting it for completeness, but all VMs, including the affected one, live on a separate ZFS mirror on the NVMe drives, not on this pool.
- 2x PCIe NVMe PM1733a (1.9 TB), NOT behind the PERC, as a ZFS mirror (R1-NVMe). This is where all VM disks live.
- BIOS: Dell 2.10.1 (04/01/2026), latest available.
Software stack
Code:
proxmox-ve: 9.2.0 (running kernel: 7.0.14-5-pve)
pve-manager: 9.2.4
pve-qemu-kvm: 11.0.2-1
intel-microcode: 3.20260227.1 (CPU running rev 0x2b000670, see microcode note)
machine: pc-q35-11.0Guest config (clean repro VM is essentially identical)
Code:
agent: enabled=1
bios: ovmf
boot: order=scsi0;ide0
cores: 16
cpu: host
machine: pc-q35-11.0+pve1
memory: 32768
ostype: win11
scsi0: R1-NVMe:vm-104-disk-1,discard=on,iothread=1,size=120G,ssd=1
scsihw: virtio-scsi-single
vga: qxlWhat I've checked on the host, all clean
ECC / memory (EDAC), 0 errors on all controllers:
Code:
/sys/devices/system/edac/mc/mc*/ce_count -> all 0
/sys/devices/system/edac/mc/mc*/ue_count -> all 0
edac-util: No errors to report.No machine-check / hardware errors:
Code:
journalctl -k | grep -iE "edac|mce|hardware error|machine check" -> nothing relevantNVMe SMART clean (0 media errors, 0 error-log entries, temps normal). ZFS pools ONLINE, 0 read/write/checksum errors. iDRAC hardware inventory shows all CPUs, DIMMs, and the PERC/battery healthy.
The key part: when the guest hangs, the host kernel log shows nothing at all. No OOM, no split-lock, no KVM error, no segfault, no soft lockup. The only VM-related lines are the tap interface going up/down from my own manual restarts:
Code:
kernel: tap104i0: left allmulticast mode
kernel: vmbr0: port 4(tap104i0) entered disabled state
kernel: tap104i0: entered promiscuous mode
kernel: vmbr0: port 4(tap104i0) entered forwarding stateGuest side
Windows only logs the aftermath, Event 41 (Kernel-Power, "rebooted without cleanly shutting down") and 6008 (unexpected shutdown), which I believe are just the result of my own forced stop, not a spontaneous bugcheck. No MEMORY.DMP, no minidump, no BugCheck (1001) event, consistent with a hang rather than a Windows crash. I corrected a bad CrashDumpEnabled value (was 7, now 3/minidump) so a real bugcheck would be captured, but it just hangs silently, which wouldn't dump anyway.
Note: kernel cmdline has split_lock_detect=off, but the kernel reports Unknown kernel command line parameters "split_lock_detect=off", will be passed to user space, so it isn't taking effect that way. kernel.split_lock_mitigate is set to 0 via sysctl. Either way, no split-lock traps appear when it hangs.
Microcode (checked across all channels, already current, ruling it out)
The kernel logs "x86/CPU: Running old microcode" and the bugs line shows old_microcode + vmscape, so I initially suspected a microcode errata. But the CPU is already on the newest microcode that exists for this signature: running rev 0x2b000670, and Intel's latest release (microcode-20260512) still lists 0x2b000670 as the current revision for 06-8f-08/87 (platform ID 0), i.e. nothing newer exists for this stepping. I understand some MCUs ship only via OEM BIOS and never in the OS package, so I also confirmed the Dell BIOS (2.10.1, latest) currently delivers the same 0x2b000670. Both the OS package and BIOS are maxed out, there's no newer microcode to apply from either channel today. Flagging it in case the "old microcode" warning itself is meaningful, but I don't believe microcode is the cause.
Important cross-check
I run the same Proxmox + Windows Server setup on older Dell PowerEdge R640 hosts (Cascade Lake Xeons) with zero issues on any Windows version. The problem only appears on this newer R760xs (Sapphire Rapids, Xeon Silver 4509Y). The guest config is identical across both, so this points at a platform/kernel interaction specific to the newer CPU rather than the VM config. I can't pin the exact date, but my recollection is that the hangs started roughly a month ago, around the time I moved to the 7.x kernel branch.
Config changes implemented so far (currently testing for stability)
- cpu: host changed to x86-64-v3 (removing raw CPU feature passthrough to the guest, my current best guess)
- vga: qxl changed to std (console black-screen made me suspect QXL)
- Guest-side sleep/hibernate disabled
Interim workaround (symptom only, not a fix)
Because the hangs mostly happen during business hours, I've deployed the community "Monitor-All" script (ping-instances), scoped via tag to only the affected VM, so it auto-resets that one guest when its agent stops responding.
Something I'm considering but haven't done yet
Rolling the host back to a 6.x kernel to test whether this is a 7.x-kernel regression on this platform. Before I take the host down for that, I'd appreciate opinions: is a 6.x rollback worth trying here, or is there a better/faster way to isolate whether this is kernel vs. cpu-model vs. QEMU machine version? I'd rather not change several variables blindly.
Questions
- Anyone seen silent Windows guest hangs (frozen console, no host log) on Sapphire Rapids Xeon with cpu: host on recent 7.x kernels?
- Best way to capture why a KVM guest wedges when the host logs nothing? I was going to attach qm monitor -> info status / info registers during a live hang to see if the vCPU is stuck, is that the right approach, or is there something better ?
- Is pc-q35-11.0 + ostype: win11 on Server 2022 a known-good combo, or should I be on a different machine type?
Happy to provide any additional logs or config. Thanks in advance.
Code:
proxmox-ve: 9.2.0 (running kernel: 7.0.14-5-pve)
pve-manager: 9.2.4 (running version: 9.2.4/5e5ae681198514d4)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-5-pve-signed: 7.0.14-5
proxmox-kernel-7.0: 7.0.14-5
ceph-fuse: 19.2.3-pve2
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20260227.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.17
libpve-guest-common-perl: 6.0.4
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.6
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.3-1
proxmox-backup-file-restore: 4.2.3-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.6
pve-cluster: 9.1.6
pve-container: 6.1.11
pve-docs: 9.2.3
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-5
pve-ha-manager: 5.2.4
pve-i18n: 3.9.0
pve-qemu-kvm: 11.0.2-1
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.0
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.3-pve1