Cross-referencing thread 183981, which describes the same cascade. We hit what looks like a related — but not identical — failure and can add a second site, a newer version, and clean hardware data.
Environment
What happened
On a single node, pve-ha-lrm segfaulted, which stopped the watchdog from being updated; the softdog reset the node ~60s later, and the CRM master then fenced it and recovered the guests. The cascade (local time, CEST):
12:36:52 kernel: pve-ha-lrm[2298]: segfault at c ip ... error 4 in perl[...] likely on CPU 38
12:36:52 watchdog-mux[1646]: client (PID 2298) did not stop watchdog - disable watchdog updates
12:36:52 systemd: pve-ha-lrm.service: Main process exited, code=killed, status=11/SEGV
12:36:53 kernel: watchdog: watchdog0: watchdog did not stop!
~12:37 node hard-reset by softdog
The CRM master saw the node go online => unknown at 12:37:33, transitioned services to fence at 12:38:23, acquired the fence lock and recovered the guests. Node was back ~12:40. All expected HA behaviour — the node reset itself because the LRM died, not the other way around.
The interesting part
This resembles 183981, but the warning location differs. In 183981 the "Attempt to free unreferenced scalar" warning was in LRM.pm. In our case no such warning appeared for the LRM at all. Instead, ~10h earlier, the same perl refcount error fired in a different daemon:
02:39:04 pvestatd[2183]: Attempt to free unreferenced scalar: SV 0x...,
Perl interpreter: 0x... at /usr/share/perl5/PVE/Status/OpenTelemetry.pm line 387.
So on this one node, on this one day, we got two independent perl heap-corruption events in two unrelated daemons (pvestatd via the OpenTelemetry status plugin, and pve-ha-lrm), both ending in a "free unreferenced scalar" / SIGSEGV class of fault. The OTel metric export is a cluster-wide setting, yet the OpenTelemetry.pm:387 warning fired only on this node.
What we ruled out
Open questions
Will continue to monitor.
Environment
- PVE 9.2.3, pve-ha-manager 5.2.4, libpve-common-perl 9.1.13
- 11-node cluster, HA-enrolled, HP hardware
- (full pveversion -v below)
What happened
On a single node, pve-ha-lrm segfaulted, which stopped the watchdog from being updated; the softdog reset the node ~60s later, and the CRM master then fenced it and recovered the guests. The cascade (local time, CEST):
12:36:52 kernel: pve-ha-lrm[2298]: segfault at c ip ... error 4 in perl[...] likely on CPU 38
12:36:52 watchdog-mux[1646]: client (PID 2298) did not stop watchdog - disable watchdog updates
12:36:52 systemd: pve-ha-lrm.service: Main process exited, code=killed, status=11/SEGV
12:36:53 kernel: watchdog: watchdog0: watchdog did not stop!
~12:37 node hard-reset by softdog
The CRM master saw the node go online => unknown at 12:37:33, transitioned services to fence at 12:38:23, acquired the fence lock and recovered the guests. Node was back ~12:40. All expected HA behaviour — the node reset itself because the LRM died, not the other way around.
The interesting part
This resembles 183981, but the warning location differs. In 183981 the "Attempt to free unreferenced scalar" warning was in LRM.pm. In our case no such warning appeared for the LRM at all. Instead, ~10h earlier, the same perl refcount error fired in a different daemon:
02:39:04 pvestatd[2183]: Attempt to free unreferenced scalar: SV 0x...,
Perl interpreter: 0x... at /usr/share/perl5/PVE/Status/OpenTelemetry.pm line 387.
So on this one node, on this one day, we got two independent perl heap-corruption events in two unrelated daemons (pvestatd via the OpenTelemetry status plugin, and pve-ha-lrm), both ending in a "free unreferenced scalar" / SIGSEGV class of fault. The OTel metric export is a cluster-wide setting, yet the OpenTelemetry.pm:387 warning fired only on this node.
What we ruled out
- Corosync / quorum flap: no. It was a process segfault, not a ring loss — corosync only showed the node leaving after the reset.
- Storage stall: no blocked-task / txg_sync / iSCSI errors in the previous boot around the event.
- Version skew from a rolling upgrade: no — the node is byte-identical to its peers (same pve-ha-manager / pve-manager / libpve-common-perl).
- RAM/CPU hardware: no evidence. Kernel EDAC/MCE clean; a full year of HPE iLO AHS shows no memory/CPU/thermal/ASR/NMI events and component health OK throughout — the only entry for the day is a plain "Server reset". (Not proof of healthy HW, but two independent layers see nothing.)
- On-disk package corruption: debsums clean across the perl + libpve-common + pve-ha-manager stack.
Open questions
- Has anyone seen "Attempt to free unreferenced scalar" across multiple perl daemons on the same host? That pattern feels like it points below the individual plugin code — a shared XS module or the interpreter — rather than a bug isolated to LRM.pm.
- Is PVE/Status/OpenTelemetry.pm:387 a known refcount issue? (Line number is for the 9.1.13 libpve-common-perl build; happy to paste the surrounding code.)
- Is the LRM.pm fault in 183981 possibly the same underlying cause surfacing in whichever daemon happens to be active?
Will continue to monitor.
Code:
~# pveversion -v
proxmox-ve: 9.2.0 (running kernel: 7.0.14-4-pve)
pve-manager: 9.2.3 (running version: 9.2.3/d0fde103346cf89a)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-4-pve-signed: 7.0.14-4
proxmox-kernel-7.0: 7.0.14-4
proxmox-kernel-7.0.6-2-pve-signed: 7.0.6-2
proxmox-kernel-6.17: 6.17.13-15
proxmox-kernel-6.17.13-15-pve-signed: 6.17.13-15
proxmox-kernel-6.17.13-13-pve-signed: 6.17.13-13
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
proxmox-kernel-6.14: 6.14.11-9
proxmox-kernel-6.14.11-9-pve-signed: 6.14.11-9
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
proxmox-kernel-6.14.11-2-pve-signed: 6.14.11-2
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-11-pve-signed: 6.8.12-11
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
amd64-microcode: 3.20251202.1~bpo13+1
ceph-fuse: 19.2.3-pve4
corosync: 3.1.10-pve2
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve2
ifupdown2: 3.3.0-1+pmx12
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.1
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.1.13
libpve-guest-common-perl: 6.0.3
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.6
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.5
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-1
proxmox-backup-client: 4.2.1-1
proxmox-backup-file-restore: 4.2.1-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.3
pve-cluster: 9.1.6
pve-container: 6.1.10
pve-docs: 9.2.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.18-4
pve-ha-manager: 5.2.4
pve-i18n: 3.7.5
pve-qemu-kvm: 11.0.0-4
pve-xtermjs: 6.0.0-1
qemu-server: 9.1.16
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.3-pve1