[SOLVED] No quorum all 3 nodes

[bensode]

We seemed to have lost quorom on all three nodes running 6.0.5 overnight! The guests are running, ceph storage is available but now each node thinks the other two are offline. I'm unable to move guests, backup, restore or make config changes because I get permission denied. How can I rescue the VMs?

 

[Stoiko Ivanov]

check the journal for possible reasons - most likely it will come from corosync and pve-cluster:
* `journalctl -u corosync -u pve-cluster`

* maybe restarting corosync on all nodes could help...

hope this helps!

 

[bensode]

Thank you! restarting corosync brought up 2 of the three hosts back into the cluster and gave quorum. I'm able to move guests disks out of the ceph cluster and backup guests while we troubleshoot the hardware issue on the third node.