4 nodes setup with 3 nodes Ceph, quorum votes 2-2-2-1 possible?

[norbertm]

Dear community,

I would like to add one additional node to an existing 3-nodes HA+Ceph cluster setup.
I know recommendation to use a QDevice for 4-nodes setup as that would allow for 2-nodes failure --> higher availablilty.
But as only 3 designated nodes host the Ceph OSDs, only one of those 3 may fail.

My idea is to run a 2-2-2-1 votes configuration for the nodes by editing /etc/pve/corosync.conf and setting "quorum_votes: 2" for all 3 Ceph nodes.
The additional/new node would get quorum_votes: 1 as usual.
So ending up with "Expected votes: 7" and "Quorum: 4"

Would this work? Safe?

PS: With 5 nodes I would use 3-3-3-1-1 ...

 

[norbertm]

Was this a stupid question? No one to answer?

 

[UdoB]

Was this a stupid question?

No! But possibly a complex one - and as usual: it depends.

You know already that you need to differentiate between PVE Quorum and Ceph Majority. This is crucial and your statements seem right to me.

With 5 nodes I would use 3-3-3-1-1 ...

This results in Expected=11 and Quorum=6. So yeah, two of the main machines should be able to maintain active state.

But then... in that situation three of five machines are dead. Do you think it is OK for the cluster to lie to itself "everything's fine"? I am not sure about this one.


That said... I need to admit that I do it similar (in my home-lab, not at my professional day job). My three main nodes have two votes. (Some other machines are turned off most of the time, due to the crazy energy costs in Germany.)

There's more than one way to skin a cat, you know --> do it and test it

 

[norbertm]

Thanks for your comments!
So I will go for it and give it a try.
I was afraid to change votes settings, without confirmation in a production system, as in most forum posts Proxmox staff and other experienced people explicitly warn against changing default voting settings.
Was afraid that it might be untested/buggy with uncommon settings...

But then... in that situation three of five machines are dead. Do you think it is OK for the cluster to lie to itself "everything's fine"? I am not sure about this one.

Only a small portion of the VMs are configured for HA. The majority of the VMs are tied to explicit hosts and are being moved manually only during updates, maintenance, etc.

... due to the crazy energy costs in Germany.

Balkonkraftwerk + Batteriespeicher zu Hause Von Mai bis Oktober kommt der Strom vom Dach des Gartenhäuschens

 

[Neobin]

I was afraid to change votes settings, without confirmation in a production system, as in most forum posts Proxmox staff and other experienced people explicitly warn against changing default voting settings.
Was afraid that it might be untested/buggy with uncommon settings...

If your subscriptions are "basic" or higher, you should ask the Proxmox support directly, if such a setup is (still) covered by their support via the support tickets...