Sudden Loss of Quorum ProxMox 6.3

[vectorhex]

We have a cluster of 13 hosts all running Proxmox 6.3. This morning all hosts have lost quorum. We can still ssh into all hosts and all hosts can ping one another. This cluster has been up and running for about a month with no issues. We have noticed a large increase in traffic on our switches that connect all the hosts together since early this morning. We have tried restarting corosync and pve-cluster services on all nodes but this has not helped.

Output from syslog is attached from three of the nodes.

Anyway to gain quorum again?

Thanks.

 

[Dominic]

What is the output of

pvecm status

? Do you have a separate corosync network?

 

[vectorhex]

Thank you for the quick response.

No we do not have a separate corosync network.

After looking at syslog on a few of the nodes it was clear that the corosync service was generating the traffic. I was able to remedy the issue by stopping the corosync service on all nodes. This immediately stopped the increase in traffic. As for getting all nodes to form quorum again, these were the steps I took:

• Pick two nodes and bring up corosync service
• Set pvecm expected to two on both hosts
• Allow those nodes to join, happened almost instantly
• Access the replication jobs of the two nodes and put a rate limit on them
• Wait for any replications to finish
• Start corosync on next node and set pvecm expected to three
• Wait for node to join
• Access replication jobs on this node and put a rate limit on them
• Wait for any replications to finish
• Repeat with remaining nodes.

Cluster is back up and quorate once again. Traffic on switches has slowed as well.

I believe, as you mentioned, that a separate network for corosync traffic would have prevented loss of quorum.

Again, thank you for the response.

 

[Dominic]

Great that you could solve it!

It is really important for Corosync to have low latencies. It is bad if your network is congested and the latency increases. This is why we recommend a separate network for it. See Cluster Network Requirements section of the documentation for some more details.

 

[mailinglists]

Or use vlans with QoS.