[SOLVED] Quorum failure and recovery

D

dafydd

Well-Known Member
Apr 24, 2010
51
7
73
Cardiff, Wales
I have a 4 node cluster running Proxmox 9. I just updated one node with the latest packages and kernel. I spotted something odd and realised that the cluster had no quorum. I don't know if that was due to the upgrade or whether it happed previously (I suspect it did). I rebooted the first node but I can't restart the container on it.

I can't risk restarting the other 3 nodes if the containers won't start up.

The nodes are all there in /etc/corosync/corosync.conf and on the same config version.

How can I recover the cluster or at least put them into a state where I would start the containers?
 
These are the errors showing on the 3 other nodes

pvescheduler[1505772]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
pvescheduler[1505771]: replication: cfs-lock 'file-replication_cfg' error: no quorum!

On the node I updated, just lots of this
corosync[2837]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
 
All nodes left the cluster in the early morning

Oct 27 03:53:51 corosync[1602]: [QUORUM] Sync members[1]: 2
Oct 27 03:53:51 corosync[1602]: [QUORUM] Sync left[3]: 1 3 4
Oct 27 03:53:51 corosync[1602]: [TOTEM ] A new membership (2.ba4) was formed. Members left: 1 3 4
Oct 27 03:53:51 corosync[1602]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4
Oct 27 03:53:51 corosync[1602]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Oct 27 03:53:51 corosync[1602]: [QUORUM] Members[1]: 2
Oct 27 03:53:51 corosync[1602]: [MAIN ] Completed service synchronization, ready to provide service.

followed by
Oct 27 03:55:32 corosync[1602]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
Oct 27 03:55:32 corosync[1602]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
 
Last edited:
Hi @dafydd ,

Absent the quorum your /etc/pve is likely Read Only, which would prevent your containers from starting.

The path below carries a certain risk and using it without proper full understanding is your call.

On one node only run: pvecm expected 1
This will allow that node to bypass proper quorum and be able to serve VMs/Containers. Then continue your investigation. Specifically "send local failed" points to a network issue (MTU, link, etc). Do not make any changes (guests or storage), Keep in mind that this is a temporary workaround, i.e. its not persistent.

Beyond that, keep investigating your journalctl, confirm the networking is well, etc. Troubleshooting failed cluster via forum back and forth is a time consuming effort. If you need fast recovery, you should engage your PVE Support Subscription (if you have one), or a PVE partner if this is business-affecting.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hmm it recovered by itself just about 24 hours later. I don't know if that was due to processes restarting or whether it was a networking issue. The local VLAN was working the whole time because the nodes could all see the NFS storage on another node. I was planning on asking the hosting provider if anything changed with their switches.
 
For anyone reading this in future, this issue coincided exactly with the backup job starting at 4am (and then running again the next day which recovered quorum). The backup failed on the 27th but worked the following night. Never seen that before in 15 years of running Proxmox,
 
Using the built-in backup scheduler to back up all containers to a node in the cluster which runs an NFS server. There was no backup process in the log for 27 October. I did get a brief ping monitoring outage for one node, which would have been the first node in the backup list.
 
You must log in or register to reply here.
Top Bottom
Back