Setting Up a Dynamic Proxmox Cluster with Normally-Off Nodes (Non-HA Setup)

[Roo42]

Hello Proxmox Community,

I'm working on a home lab and need to save power, so most of my nodes are powered off most of the time. My goal is to set up a Proxmox cluster that can dynamically scale as nodes are powered on and off without requiring high availability (HA) or strict quorum. I’m fine with reduced availability if it means saving on energy costs.

Main Goal​

I’d like to establish a “dynamic” cluster with one primary node that’s always on, while other nodes can stay off until needed. Flexibility in handling quorum with minimal power usage is my top priority here.

Specific Questions​

• Quorum with Dynamic Nodes: I know it’s possible to adjust quorum settings to allow a single-node vote, but I’ve read that this can cause issues. Could anyone clarify the possible challenges with this approach, and suggest any workarounds that might help avoid these?
  
  
• Minimal Always-On Nodes for Quorum: To maintain a low-power quorum, I’ve considered adding two Raspberry Pis so that 3 nodes would always be online. However, I’m unclear on whether this would only work if these 3 nodes hold 50% or more of the voting power. Does anyone know the required vote percentage for an always-on quorum, and if I’d need to adjust vote settings as other nodes power on and off?
  
  
• Best Practices: Are there any recommended configurations or best practices for managing a cluster where nodes are frequently turned on and off?

Any insights into optimizing Proxmox clusters for low-power, dynamic configurations would be greatly appreciated!

Thank you,
Root42

 

[Moayad]

Hi,

If you want to avoid quorum loss, consider installing QDevice [0] on a Raspberry Pi to maintain quorum with minimal power usage.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support

 

[Roo42]

hi Moayad.
thanks for the answer do you mind me asking how this compares to options like last_man_standing and auto_tie_breaker ?

 

[esi_y]

They will not want to talk about those votequorum options.

Proxmox do not "support" those, i.e. they will not provide you support if you run into problems with them.

The question is ... will you run into problems with them? I am glad someone actually did their own research and found those options.

Of course they work! But you need to know what you are doing and the implications, e.g. you will have to be able to accomodate for WFA after complete shutdown.

 

[t.lamprecht]

thanks for the answer do you mind me asking how this compares to options like last_man_standing and auto_tie_breaker ?

Compared to those two options a QDevice is much more reliable as it's an actual third external voter, the other only work in some more specific cases, like auto_tie_breaker only works for a specific node failure as it automatically gives the node with the lower (or higher, can be configured) ID the remaining vote, so if the node breaks that would get that vote the other one won't have qurorum, with a QDevice it still could get quorum.

 

[esi_y]

Compared to those two options a QDevice is much more reliable

But the OP talks of adding extra devices to have 3 votes and stay quorate, so I extrapolate he has 5 nodes? There's no way he can dynamically scale down quorum with QD alone, except there's options for the QD that are along similar lines:
https://manpages.debian.org/bookworm/corosync-qdevice/corosync-qdevice.8.en.html

 

[esi_y]

@Roo42 If you are doing LMS with QD (which actually is a nice setup), consider the standard PVE tooling is doing way more than is discoverable at first:
https://forum.proxmox.com/threads/q...-discourage-from-perfectly-safe-setup.139809/

 

[t.lamprecht]

There's no way he can dynamically scale down quorum with QD alone

Why not? It certainly provides enough votes for that.

except there's options for the QD that are along similar lines:

If you refer to "tie_breaker" then they may sound similar but cannot be compared at all to the non-QD ones, as the latter is an external third vote that can observe all quorum partitions and thus be a real and safe arbiter. The non-QD ones will always just work in some specific cases or be unsafe.

Quorum device were developed as evolution on the old tiebreakers and to fix their inherit design issues.

 

[esi_y]

Why not? It certainly provides enough votes for that.

Yes my bad, I almost forgot the PVE defaults to ... not the default in this case.

If you refer to "tie_breaker" then they may sound similar but cannot be compared at all to the non-QD ones, as the latter is an external third vote that can observe all quorum partitions and thus be a real and safe arbiter. The non-QD ones will always just work in some specific cases or be unsafe.

But I talk in the framework of the OP where he is not going to use HA. If he has e.g. only 2 nodes, there's absolutely nothing wrong with setting two_node:1 except the OP must take note that it means wait_for_all: 1 as well. I cannot think of anything (without HA) where the extra QD would give him anything extra (and he has to run extra device).

Quorum device were developed as evolution on the old tiebreakers and to fix their inherit design issues.

But you strongly recommend it mostly because you always assume HA will be used.

 

[t.lamprecht]

wait_for_all: 1

This is a disadvantage if one want's to do maintenance on the single node left, and e.g. simply rebooting into a new kernel, as without the other online after the reboot it will never get quorate again.

I cannot think of anything (without HA) where the extra QD would give him anything extra (and he has to run extra device).

See above sentence and also OP talks about using Raspberry Pi's for extra nodes, running full-blown PVE (or even just a manually reduced/maintained corosync stack) causes much higher power use and wear out of the Pi compared to the very simple and much less resource using QDevice that only has to reply on update requests every 20s or if a quorum partition changes. Further, the Pi would need to participate in the totem/cpg messages without any benefit of doing that.
If any extra device is used just for sake of quorum it will always be better to integrate it as qdevice, not full cluster node member.
And without QDevice LMS and Co always have edge case that depending on configuration will simply fail quorum where QD wouldn't, or worse, cause a split-brain and potential data or service corruption.

 

[esi_y]

This is a disadvantage if one want's to do maintenance on the single node left, and e.g. simply rebooting into a new kernel, as without the other online after the reboot it will never get quorate again.

No no, taking my premise that we are talking e.g. only 2 nodes - this is suggesting running 3rd device so that ... he does not have to artifically be setting quorum to 1 (when he knows the other one is off). That's the only benefit? Then he is dependent on that QD to be up.

See above sentence and also OP talks about using Raspberry Pi's for extra nodes, running full-blown PVE (or even just a manually reduced/maintained corosync stack) causes much higher power use and wear out of the Pi

Yes that's terrible idea.

compared to the very simple and much less resource using QDevice that only has to reply on update requests every 20s or if a quorum partition changes. Further, the Pi would need to participate in the totem/cpg messages without any benefit of doing that.

Correct.

If any extra device is used just for sake of quorum it will always be better to integrate it as qdevice, not full cluster node member.

Yes. But none of the above sells me QD (vs nothing).

And without QDevice LMS and Co always have edge case that depending on configuration will simply fail quorum where QD wouldn't, or worse, cause a split-brain and potential data or service corruption.

Some people just set one node e.g. 2 votes and call it their master. I happen to prefer have e.g. auto tie breaker set. The implicit WFA for two_node:1 is there mostly to avoid disaster in HA scenarios.

 

[t.lamprecht]

No no, taking my premise that we are talking e.g. only 2 nodes - this is suggesting running 3rd device so that ... he does not have to artifically be setting quorum to 1 (when he knows the other one is off). That's the only benefit? Then he is dependent on that QD to be up.

Besides that something that automatically works is always nicer compared to having to remember actively doing, as that can mean many hours of services lost (bad if e.g. ones NVR runs on this), it's just one example, if power is lost and restore-power state is configured to "last state" then it would help too, and as said the whole split-brain possibility on misconfiguration makes it simpler the safer option, if you do not need that then fine, but do not advertise options that can have a dangerous impact in the hands of less experienced people as better.

QDevice will always be the safer option that was explicitly designed as response to many users holding the previous ways wrong and causing them a broad spectrum of issues, from barely annoying to actual data loss; OP knows best about the importance of their setup's services and data and what route to take, if they want safer and do not care about an extra running device, which might run anyway, we do not know that, then a QDevice it is, if they do not want an extra device at all and are fine with any possibly implications, whyever that is, then they can use none; that still doesn't mean one needs to run around and advertise that as better blanket option.

Yes. But none of the above sells me QD (vs nothing).

But this is not your thread, so lets not again further hijack it with your premises and let OP actually read the responses and ask potential follow-up questions for their use cases, without having to wade through a dozen or so of tangentially related replies and overly specific nitpicking.

 

[esi_y]

do not advertise options that can have a dangerous impact in the hands of less experienced people as better.

This was OP already coming with the homework done, actually.

QDevice will always be the safer option

This depends on the definition of "safe" - I suppose safe for split brain and all the things that matter for HA. For me the fewer machines that can break is also "safer" setup (in a different sense).

that was explicitly designed as response to many users holding the previous ways wrong and causing them a broad spectrum of issues, from barely annoying to actual data loss;

Data loss on split brain without HA?

 

[t.lamprecht]

Data loss on split brain without HA?

I.e., "data loss can happen if all guards and handrails protecting from that are removed? I'm shocked..."

Yes, if the protections that make using clustering and the PVE API actually safe are willing disabled as standard part of one's workflow, compared as manual fix to resolve an unexpected situation after closely checking, e.g. with our support, and ensuring doing so is fine.

But again, stop being so disrespectful to other users by hijacking every thread to make it about you and having staff educate you for free, as already said:

But this is not your thread, so lets not again further hijack it with your premises and let OP actually read the responses and ask potential follow-up questions for their use cases, without having to wade through a dozen or so of tangentially related replies and overly specific nitpicking.

 

[esi_y]

I.e., "data loss can happen if all guards and handrails protecting from that are removed? I'm shocked..."

Yes, if the protections that make using clustering and the PVE API actually safe are willing disabled as standard part of one's workflow, compared as manual fix to resolve an unexpected situation after closely checking, e.g. with our support, and ensuring doing so is fine.

But again, stop being so disrespectful to other users by hijacking every thread to make it about you and having staff educate you for free, as already said:

This is false information (without being concrete enough to show how) for the OP. His premise is HA is off. Given that, talking of data loss in a situation where worst outcome is inquorate nodes? Or I do not know something, then educate us all...

 

[Azunai333]

This is false information (without being concrete enough to show how) for the OP. His premise is HA is off. Given that, talking of data loss in a situation where worst outcome is inquorate nodes? Or I do not know something, then educate us all...

The OP may know the consequences of driving without a seat belt.
But this is an open forum with new users which will read this, too. And they may not know the consequences of driving a car without a seat belt. Just because you drive without one doesn't mean something bad happens (immediately).

Those new users may not know what's stupid to do and what will lead to data loss. For example, you could try to move the VM config file to another node, then start the VM a second time on the other node while it is still running on the first node: Hello data loss and/or split brain. For those users this may lead to problems later.

The OP has HA not active, but this only means it doesn't explode right away. Effectively you'll increase the risk of data loss if something other is gonna happen. But you have to know this (and don't come crying to the forum later). @t.lamprecht is just trying to make that clear. Nobody wants data loss.

I wasn't sure if I should answer this, because we should really go back to topic now.

 

[esi_y]

The OP may know the consequences of driving without a seat belt.

Whether OP uses the votequorum options or QD, he will not have quorum when/where he should not have it. He just needs to use votequorum options correctly.

But this is an open forum with new users which will read this, too.

This would mean I cannot reply to anyone on their specific question (note my reply came after their follow up question only).

The OP has HA not active, but this only means it doesn't explode right away. Effectively you'll increase the risk of data loss if something other is gonna happen.

This was my issue early on that PVE does not provide a foolproof way to disable HA (and watchdogs). I can help the OP further to secure himself from the HA stack kicking in should there be any bugs.

I wasn't sure if I should answer this, because we should really go back to topic now.

I think anyone should be free to answer, I just do not like to be told to stop disputing the "official" answer. If everyone comes here for those, then this could be ChatGPT fed with wiki.