Cluster Behavior

[bellaireroad]

I have recently setup a cluster on 4 machines - X11, X11A, X12, and H13. I have H13 offline at present.
X11 is running a couple of containers and a VM. Last night, X11A went offline, not sure why. I logged into the IPMI, and proxmox was running, although I could not log in via SSH.

The containers and the VM on X11 shut down.

Here is an exerpt from the log file, at the time I am guessing this happened. I did not discover this until about 11PM

Oct 17 09:32:03 X11 pmxcfs[1596]: [quorum] crit: quorum_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
Oct 17 09:32:03 X11 pmxcfs[1596]: [quorum] crit: can't initialize service
Oct 17 09:32:03 X11 pmxcfs[1596]: [confdb] crit: cmap_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
Oct 17 09:32:03 X11 pmxcfs[1596]: [confdb] crit: can't initialize service
Oct 17 09:32:03 X11 pmxcfs[1596]: [dcdb] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
Oct 17 09:32:03 X11 pmxcfs[1596]: [dcdb] crit: can't initialize service
Oct 17 09:32:03 X11 pmxcfs[1596]: [status] crit: cpg_initialize failed: CS_ERR_LIBRARY (failed to connect to corosync)
Oct 17 09:32:03 X11 pmxcfs[1596]: [status] crit: can't initialize service

Question #1 - is it expected for containers and clusters to shut down on a 4 node cluster when one of the nodes is already offline, and another node goes down unexpectedly. I don't understand why the VM and containers shut down on X11, and no migration was required when X11A went down

Observation #1 - X11 is the login for the cluster. The system log for X11A cannot be read from the cluster interface. It pulls up nothing. However, the system log can be read on X11A when logged in directly.

Observation #2. I previously created a certificates for all the machines in the cluster. The certificate for X11A is apparently no longer working. I get the unsafe warning when logging in

Any idea what caused this?

 

[beisser]

the first issue is likely related to quorum. if you have an even number of nodes and half of them go down, then quorum is at 50%, which is insufficient as more than 50% is required.
as a result the surviving nodes fence themselves, leading to the result you are seeing.

my advice would be to add something like a raspi or other single board computer and install a linux on it plus the package qdevice.
this makes it act as a 5th device for the quorum. this will prevent split brain scenarios even if 2 of your nodes go offline because the remaining 2 nodes plus the qdevice will equal 60% of quorum votes.

no idea about the certificate issue though