Why is my cluster split-brain so stable and how to fix it?

Sep 13, 2022

today my script that reconfigures 80 VMs with 80 concurrent pvesh create commands got stuck. At this time, one node was offline or right back online since a few minutes due to a little "ohh what kind of cable is this" issue of a team mate.

After using the script successfully several times, this time (all?) pvesh commands hung, as well as the web GUI. I obseverd a "100% CPU" corosync (in strace, I see heaps of epoll_wait(4, [], 12, 0) = 0 calls without delay (according to strace, a call each ~ 0.015ms). Not having a better idea I first restarted corosync on the node I accidentally was logged in anyway, which reduced CPU load (at least for a moment, didn't check after). I noticed that instead of the expected 12 node cluster I'm facing 2 x 6 node clusters... I thought it would be easy to break this deadlock by simply rebooting any node: the ring it was in would have a node less, and all its nodes will join to the now bigger ring and everything would be fine.

In contrast I observed that after the node rebooted, it found it way back in its old ring and gain both contained 6 nodes. I repeated booting with the same effect. I power-cut a node to now really force something, but after reboot again I had 2 x 6 node rings.

So I rebooted every node of one of the rings.
I dare to say that now I'm facing four rings (1 with 1 node, 2 with 3 nodes, 1 with 5 nodes)...

How to break a split-brain? Or better: how to rejoin nodes into single one?

After some jobs are supposed to be done tonight I will forcibly reboot every node at the same time in the hope that this works, but it feels wrong.

Where can I learn more about the ring management?
Last edited:


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!