Quorum failure and recovery

D

dafydd

Well-Known Member
Apr 24, 2010
48
7
73
Cardiff, Wales
I have a 4 node cluster running Proxmox 9. I just updated one node with the latest packages and kernel. I spotted something odd and realised that the cluster had no quorum. I don't know if that was due to the upgrade or whether it happed previously (I suspect it did). I rebooted the first node but I can't restart the container on it.

I can't risk restarting the other 3 nodes if the containers won't start up.

The nodes are all there in /etc/corosync/corosync.conf and on the same config version.

How can I recover the cluster or at least put them into a state where I would start the containers?
 
These are the errors showing on the 3 other nodes

pvescheduler[1505772]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
pvescheduler[1505771]: replication: cfs-lock 'file-replication_cfg' error: no quorum!

On the node I updated, just lots of this
corosync[2837]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
 
All nodes left the cluster in the early morning

Oct 27 03:53:51 corosync[1602]: [QUORUM] Sync members[1]: 2
Oct 27 03:53:51 corosync[1602]: [QUORUM] Sync left[3]: 1 3 4
Oct 27 03:53:51 corosync[1602]: [TOTEM ] A new membership (2.ba4) was formed. Members left: 1 3 4
Oct 27 03:53:51 corosync[1602]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4
Oct 27 03:53:51 corosync[1602]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Oct 27 03:53:51 corosync[1602]: [QUORUM] Members[1]: 2
Oct 27 03:53:51 corosync[1602]: [MAIN ] Completed service synchronization, ready to provide service.

followed by
Oct 27 03:55:32 corosync[1602]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
Oct 27 03:55:32 corosync[1602]: [KNET ] loopback: send local failed. error=Resource temporarily unavailable
 
Last edited:
Hi @dafydd ,

Absent the quorum your /etc/pve is likely Read Only, which would prevent your containers from starting.

The path below carries a certain risk and using it without proper full understanding is your call.

On one node only run: pvecm expected 1
This will allow that node to bypass proper quorum and be able to serve VMs/Containers. Then continue your investigation. Specifically "send local failed" points to a network issue (MTU, link, etc). Do not make any changes (guests or storage), Keep in mind that this is a temporary workaround, i.e. its not persistent.

Beyond that, keep investigating your journalctl, confirm the networking is well, etc. Troubleshooting failed cluster via forum back and forth is a time consuming effort. If you need fast recovery, you should engage your PVE Support Subscription (if you have one), or a PVE partner if this is business-affecting.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
You must log in or register to reply here.
Top Bottom
Back