Testing Fix / Prevention: pve-ha-manager deadlock during apt dist-upgrade (Automated Script)

M

Meister Mopper

Active Member
Jan 31, 2021
28
5
43
61
Problem Statement & Context

The Problem:
When performing system updates (apt-get dist-upgrade) on a Proxmox VE node within an HA cluster, a deadlock can occur during package processing:
  • Package Hang via Synchronous Triggers: During the update of the `pve-ha-manager` package, `dpkg` executes post-installation scripts that attempt to synchronously reload the local HA service (`pve-ha-lrm`). If dependent systemd services or underlying libraries are updated at the exact same time, `systemd`, `D-Bus`, and `dpkg` can deadlock each other.
  • Cascading Lockup of HA Resources: As soon as the local HA manager hangs, running VMs and LXC containers can no longer be gracefully stopped or migrated. Commands like `pct stop` freeze indefinitely in a queue (`Requesting HA stop...`) because the HA subsystem stops responding.
  • Tedious Manual Recovery: To recover the node, affected processes must be forcefully killed via `kill -9`, followed by forced node reboots (`reboot -f`).
My Solution:
To prevent this deadlock, the target node must be placed into HA maintenance mode **before** running `apt-get dist-upgrade` (`ha-manager crm-command node-maintenance enable <Node>`). Therefore, using Gemini 3.6 Flash and Antigravity IDE, I developed a script to automate this process.


Give it a spin & feedback welcome!

If you've ever bumped into locked HA processes or simply want a smoother, hassle-free workflow for node updates, feel free to give the script a test run in your lab or cluster!
Feedback, suggestions, or PRs are always appreciated => let me know how it works for you or if you encounter any edge cases:

Happy upgrading!

GitHub: proxmox-cluster-upgrade-script

This prompts the cluster to live-migrate all HA resources to the remaining nodes without downtime. During the upgrade, the node is completely isolated and free of active HA workloads, allowing the package updates to complete cleanly without any blockages.
 
please provide the exact package versions involved ("/var/log/apt/history.log"), the apt output ("/var/log/apt/term.log") and the journal covering the failed/hanging upgrade. this should not happen and sounds like a bug (somewhere)!
 
Hi @fabian,

here are the precise logs from node **HA-PVE-01** covering the exact update run on **July 16, 2026**:

1. Package Versions (`/var/log/apt/history.log`)


Code: 
Start-Date: 2026-07-16  10:18:31
Commandline: apt-get dist-upgrade
Install: proxmox-kernel-7.0.14-5-pve-signed:amd64 (7.0.14-5, automatic), proxmox-kernel-6.17.13-18-pve-signed:amd64 (6.17.13-18, automatic)
Upgrade: pve-firmware:amd64 (3.18-4, 3.18-5), proxmox-kernel-7.0:amd64 (7.0.14-4, 7.0.14-5), proxmox-backup-file-restore:amd64 (4.2.2-1, 4.2.3-1), proxmox-kernel-6.17:amd64 (6.17.13-16, 6.17.13-18), qemu-server:amd64 (9.1.18, 9.2.0), pve-container:amd64 (6.1.10, 6.1.11), proxmox-backup-client:amd64 (4.2.2-1, 4.2.3-1), libpve-common-perl:amd64 (9.1.16, 9.1.17)
End-Date: 2026-07-16  10:20:16


2. APT Terminal Output (`/var/log/apt/term.log`)


Code: 
Log started: 2026-07-16  10:18:31
Setting up pve-container (6.1.11) ...
Setting up proxmox-kernel-6.17 (6.17.13-18) ...
Setting up proxmox-kernel-7.0 (7.0.14-5) ...
Processing triggers for systemd (257.13-1~deb13u1) ...
Processing triggers for man-db (2.13.1-1) ...
Processing triggers for dbus (1.16.2-2) ...
Processing triggers for procps (2:4.0.4-9) ...
Processing triggers for pve-ha-manager (5.2.4) ...
Processing triggers for pve-manager (9.2.4) ...
Log ended: 2026-07-16  10:20:16


3. Journal Logs (`journalctl`)

The system journal for July 16th has already been rotated out due to retention limits, so no `journalctl` entries remain for that timeframe. However, the APT history and terminal logs above capture the exact packages and triggers involved during the run.

Please let me know if you need any additional information!
 
unfortunately without the system logs it is impossible to tell what caused the hang..
 
Hi @fabian,

I checked `/var/log/syslog*` using `zgrep` for July 16th, but unfortunately, those syslog files have also already been rotated out and deleted due to the log retention settings on this node.

Please let me know if there are any other diagnostic logs or details I can provide!
 
no, those two would be the main ones.
 
You must log in or register to reply here.
Top Bottom
Back