HA Cluster Manager stuck in wait_for_quorum after rejoining cluster node (PVE 8.4.0)

[Satyam Kushwaha]

Hello Proxmox Support,

We have a 3-node Proxmox VE 8.4.0 cluster.

Environment:
- 3 nodes
- Corosync quorum is healthy
- pmxcfs is healthy
- watchdog-mux is running on all nodes
- All nodes are running PVE 8.4.0
- No HA resources are currently configured

Issue:
One node (proxmoxtwo) was removed from the cluster and later rejoined successfully.

The cluster itself is healthy:
- pvecm status shows Quorate: Yes
- All 3 nodes are members
- VMs operate normally

However, HA is not functioning:
- pve-ha-crm and pve-ha-lrm are running on all nodes
- ha-manager status never shows proxmoxtwo
- manager_status remains empty
- CRM stays in "wait_for_quorum"
- CRM never becomes master after restarting the HA services

Troubleshooting already performed:
- Restarted pve-ha-crm and pve-ha-lrm
- Cleared stale HA state (manager_status/lrm_status)
- Verified watchdog
- Verified corosync
- Verified pmxcfs
- Verified hostname consistency
- Verified all nodes run the same PVE version

Note- This cluster was rebuilt after one node was rejoined. The cluster itself is healthy, but HA has not recovered. We have avoided making further changes and would like guidance before proceeding.

Please advise on the next steps.

 

[g.fiore]

Before assuming HA is broken — is there a chance the behavior you're seeing is actually the expected state for a cluster with no HA resources configured?

The pve-ha-manager source has explicit logic for this: if the cluster has no services configured for a number of rounds, the CRM goes idle and transitions to wait_for_quorum on purpose, and doesn't touch manager_status. See the comment in CRM.pm on GitHub. So an empty manager_status + CRM stuck in wait_for_quorum + no master ever promoted is exactly what you'd see on a healthy cluster that has nothing to manage.

Quick test to confirm: add one HA resource (even just an existing stopped VM in `stopped` state via ha-manager add vm:XXX --state stopped), wait ~60 seconds, then check ha-manager status again.

If a master gets elected and the LRMs start showing up in the output, HA is fine and the rejoin didn't break anything — it just had nothing to do.

If the CRM still doesn't promote a master with a resource configured, then you have a real issue and the next things I'd check are:

- ls /etc/pve/nodes/ on all three nodes (proxmoxtwo directory should be there with the same contents structure as the other two)
- ls /etc/pve/priv/lock/ for any stale ha_manager_lock left behind
- pvecm nodes to confirm proxmoxtwo has a consistent nodeid across the cluster (a re-add with a different nodeid than the delnode used has been reported to cause weird HA state)

Post back ha-manager status output after adding the test resource and we can go from there.

 

[Satyam Kushwaha]

Thanks, it worked!