Debian guests freeze after ~3 days on PVE 9.2.21 / kernel 7.0.2-6-pve, host unaffected, no logs before hang

K

kevin2001

Member
Dec 29, 2021
8
0
21
25
Two of my three VMs freeze after roughly 3 days of uptime. SSH,
console and qemu-guest-agent all stop responding. The host stays
up and responsive the whole time. Only qm stop + qm start recovers
them.

Setup:
- proxmox-ve 9.2.0, pve-manager 9.2.21, kernel 7.0.2-6-pve (pinned)
- AMD host, single NVMe (LVM-thin), no RAID
- Realtek RTL8168h (r8169) in a VLAN-aware bridge vmbr1
- Host on VLAN 101, guests on VLAN 104
- VM 101: Debian, VirtFusion with nested KVM guests
- VM 102: Debian 13, kernel 6.12.111, Pterodactyl panel
- VM 100: same host, same bridge, never freezes

What I observed during a hang:
- qm status reports running, qm monitor responds, screendump works
- guest kernel log ends mid-stream at 11:02:02 with no panic, no
oops, no hung task, no OOM
- the kernel timestamp on the guest console stops advancing
entirely
- alt-sysrq-s/u/b via qm monitor had no effect
- qm agent ping times out

Already ruled out:
- split lock detection: lscpu reports no split_lock_detect,
journalctl --grep split_lock is empty
- the 7.0.6 to 7.0.14 freeze reports: I run 7.0.2-6 and have it
pinned
- memory pressure: guest had 3GB of 3.8GB free
- fail2ban: 0 banned, 0 failed
- disk space: 73% used
- fstrim: last run 2 days before the hang
- NIC: ethtool -S shows 0 errors, 0 drops, 0 missed, link stable
since boot, no r8169 messages in dmesg

Already configured for the next occurrence:
- ufw logging off and kernel.printk limited (console was flooded
with UFW BLOCK lines that buried any earlier output)
- serial console on both VMs with socat logging to a file on the
host

Questions:
1. Is there anything on the host side that can stop a guest this
abruptly while leaving the qemu process healthy and leaving no
trace in the guest?
2. Are there known issues with nested virtualization on AMD in
this kernel branch that could affect a sibling VM?
3. Anything else worth capturing before I hard stop the VM next
time?
 
1. please try with the most recent kernel version
2. provide pversion -v
3. provide the VM configs
 
  • Like
Reactions: fiona
Hi,
additionally to what @fabian said, the host's journal from around the time of the issue might also be interesting.
 
fabian said:
1. please try with the most recent kernel version
2. provide pversion -v
3. provide the VM configs
Click to expand...
pveversion -v and the three VM configs below (SSH keys and
cloud-init usernames removed).

Note on memory: the host has 62GB total and 58GB is allocated to
guests (50GB to VM 101, 4GB each to 100 and 102). VM 101 runs
VirtFusion with nested KVM guests. All three use cpu: host.

Regarding the kernel: I'm currently pinned to 7.0.2-6-pve because
of the reports of VM freezes on 7.0.6 through 7.0.14. 7.0.14-22 is
installed. Is that version expected to contain a fix for those
reports, or would moving there risk introducing a second failure
mode? This is a production host.
 
You mentioned no OOM killer event entries, but I still would be curious if reducing the allocated memory to VM 101 a bit would make any difference. 58GB out of 62GB seems a bit tight.
 
Hi, I'm investigating potentially related nested Linux guest failures on AMD.
Host CPU is Ryzen 7 5700G - referring to it as L0 below.

My guest is also PV2 9.2.1 - call it L1
That guest also uses nested virtualization, and has 3 guests itself, another PVE 9.2.1, as well as Ubuntu 24 & 26 - referring to them as L2.

L0 and L1: Proxmox VE 9.2.21, kernel 7.0.14-25-pve.
L1 packages: pve-qemu-kvm 11.0.3-4; pve-edk2-firmware 4.2026.08-1.
L2 guests: Ubuntu 24.04.5 and 26.04, cpu: host, OVMF/q35, four vCPUs.

I'm writing a complex tool with a test suite against all these L2 guests, that takes about 8 hours to execute. The test suite controls VM power operations, disk-only snapshots and rollbacks through the Proxmox REST API. Test commands execute over SSH inside the L2 guests.

I have been working on the tool for about 6 weeks. I have experienced random L2 guest failures, many of which I think are related to nested virtualization. These are only the latest issue reports.

Failures occur during repeated boots and cold-checkpoint rollbacks.

L0 and L1 remain responsive; QEMU reports the affected guests as running.

Ubuntu 26.04 L2 remained black after a reset and a second 240-second wait.
Its vCPU0 spins in a loop matching OVMF WaitApWakeup; vCPU3 is at the reset vector.

Ubuntu 24.04 L2 logs "CPU2 failed to report alive state" and starts only three CPUs.
It also logs soft lockups, including in memcpy_toio/simpledrm framebuffer work.
Networking appeared around 297 seconds, after an EFI-device timeout disrupted startup.

L1 logs "KVM discovered backwards TSC" and warns that guest TSC will be unreliable.

I found no OOM or disk-full evidence.

Are there recommended KVM traces or known AMD nested-SMP issues worth investigating?
 
In my home lab, I run 2 x Ryzen based systems. One of those needs `iommu=pt` added to the kernel cmd line to work reliably.

As a safe-guard, I've done this on both of my Ryzen systems and haven't had a problem since.

Might be worth a try?
 
You must log in or register to reply here.
Top Bottom
Back