Documentation Request - Full Procedure to Recover Quorum Failure

[tcabernoch]

Hi.
I'd like directions on how to troubleshoot the fuse and corosync services. What commands are needed to validate sync? How do you check it? How do you fix it?
Further, I'd like vendor provided directions on how to recover a failed quorum. This disaster happens to everyone. Really, it needs to be addressed.

Mostly, those directions currently live in this forum.
You have to look under the first respondent who sneers a the newb in order to get to the (partial) answer. And repeat. Until you find all the info.
Can you put all that in one spot, please?

 

[tcabernoch]

This is the official bit. There's not a lot there. They don't even wish you good luck.


https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_configuration

Troubleshooting​

Issue: quorum.expected_votes must be configured​

When corosync starts to fail and you get the following message in the system log:
[...]
corosync[1647]: [QUORUM] Quorum provider: corosync_votequorum failed to initialize.
corosync[1647]: [SERV ] Service engine 'corosync_quorum' failed to load for reason
'configuration error: nodelist or quorum.expected_votes must be configured!'
[...]
It means that the hostname you set for a corosync ringX_addr in the configuration could not be resolved.

Write Configuration When Not Quorate​

If you need to change /etc/pve/corosync.conf on a node with no quorum, and you understand what you are doing, use:
pvecm expected 1
This sets the expected vote count to 1 and makes the cluster quorate. You can then fix your configuration, or revert it back to the last working backup.
This is not enough if corosync cannot start anymore. In that case, it is best to edit the local copy of the corosync configuration in /etc/corosync/corosync.conf, so that corosync can start again. Ensure that on all nodes, this configuration has the same content to avoid split-brain situations.