Does a 3 nodes cluster + a Qdevice, allows a single PVE host to continue running VMs?

[logui]

Sometimes in the 3 nodes cluster (home-lab), I have to do some hardware changes or repairs on 2 of the nodes/pve hosts, instead of doing the 2 pve host's repairs in parallel, I have to do it one at a time, to always keep two nodes up, running and connected, because If I leave only one pve host running, it will shutdown all the VMs due to lack of quorum.

I have been thinking on setting up a Qdevice on a small Raspberry Pi NAS that I have, will this configuration of 1 pve host + Qdevice allow the VMs in the pve host continue running, while I have the other 2 nodes/pve hosts temporary down for maintenance?
Thanks

 

[Daniel_Dog]

No, unfortunately.
This is really a catch 22 case.

For Proxmox VE to determine if a node is offline it uses a voting system.
If more then 50% of the votes say its offline then it is marked as offline and will migrate VMs as configured via HA.
if 2 out of 4 nodes are offline then it can only have at max 50% of the votes, and thus not satisfy the over 50% vote requirement.
This is also the reason why a 2 node configuration does not work without a extra Qdevice as voter.

But if you want to (and if you can) you could add 2 Qdevices so that even when 2 nodes are offline it still can have a greater then 50% vote count and thus work.

 

[Daniel_Dog]

And in general all clusters without a odd number of nodes has this catch 22 issue.
But when you scale up the node count is it less likely that 50% or more nodes are offline at the same time.
And thus less of a concern.

 

[logui]

No, unfortunately.
This is really a catch 22 case.

For Proxmox VE to determine if a node is offline it uses a voting system.
If more then 50% of the votes say its offline then it is marked as offline and will migrate VMs as configured via HA.
if 2 out of 4 nodes are offline then it can only have at max 50% of the votes, and thus not satisfy the over 50% vote requirement.
This is also the reason why a 2 node configuration does not work without a extra Qdevice as voter.

But if you want to (and if you can) you could add 2 Qdevices so that even when 2 nodes are offline it still can have a greater then 50% vote count and thus work.

Thank you, yes, I have 3 Raspberry Pi devices providing different functionality, therefore I can install the Qdevice service on 2 of them for the 5 nodes cluster (from the voting/quorum respective), this seems a good idea. To summarize, I can have two pve/host down, and keep only one pve/host running with the VMs while I have 2 QDevices, right?

 

[Daniel_Dog]

Yes, as long as the 2 left over nodes can take the load it gets. (CPU and Memory wise)

 

[logui]

Yes, as long as the 2 left over nodes can take the load it gets. (CPU and Memory wise)

Well, it is actually just one node (real node, not a qdevice) that will be left, and yes, the overall cluster load is very small, so, one node will run all the VMs and the 2 QDevices will help with the quorum

 

[Daniel_Dog]

And as long as it has a greater then 50% voting.
So with 3 Qdevices every node except 1 can de down.
With 2 QDevices at least 2 nodes noeed to stay online.

It all comes down to:
It needs to have enough voters to satisfy the over 50% vote requirement.

 

[leesteken]

Thank you, yes, I have 3 Raspberry Pi devices providing different functionality, therefore I can install the Qdevice service on 2 of them for the 5 nodes cluster (from the voting/quorum respective), this seems a good idea. To summarize, I can have two pve/host down, and keep only one pve/host running with the VMs while I have 2 QDevices, right?

I expect that you can only use one QDevice (as it is intended as a tie breaker for an even number of nodes).
Why not give your one special node 3 votes? Then you can shut both other nodes down. Proxmox is not designed to operate like that (and you cannot run from both other nodes without that one big node) but it'll probably work.

Sometimes in the 3 nodes cluster (home-lab), I have to do some hardware changes or repairs on 2 of the nodes/pve hosts, instead of doing the 2 pve host's repairs in parallel, I have to do it one at a time, to always keep two nodes up, running and connected, because If I leave only one pve host running, it will shutdown all the VMs due to lack of quorum.

Why not just do that: work on one node at a time. Then everything will be fine and within the intended and supported design.

 

[Johannes S]

I expect that you can only use one QDevice (as it is intended as a tie breaker for an even number of nodes).
Why not give your one special node 3 votes? Then you can shut both other nodes down. Proxmox is not designed to operate like that (and you cannot run from both other nodes without that one big node) but it'll probably work.

As a temporary workaround this could be ok. It's even the suggested way to recover when a cluster gets broken. Afterwards you should change the vote back to one of course. In any case this will be a better solution than shenangians with two qdevices.
It should also be noticed that the manual recommends AGAINST using a qdevice on clusters with an odd count of nodes since it actually increases the chance of failure instead of reducing it:
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[logui]

And as long as it has a greater then 50% voting.
So with 3 Qdevices every node except 1 can de down.
With 2 QDevices at least 2 nodes noeed to stay online.

It all comes down to:
It needs to have enough voters to satisfy the over 50% vote requirement.

I am a little confused, with a 3 node cluster plus 2 qdevices = 5, if 2 nodes are down, I will have 3 (1 host + 2 qdevices) up > 2.5 (50% of 5), please could you explain

 

[Daniel_Dog]

Sorry my mistake.
3 nodes + 2 Qdevices is correct and result in a vote that is greater then 50% and thus will work.

 

[Johannes S]

Sorry my mistake.
3 nodes + 2 Qdevices is correct and result in a vote that is greater then 50% and thus will work.

It's still an option not mentioned in the docs and which in former discussions didn't yield a lot of support for:
https://forum.proxmox.com/threads/add-2-qdevice.147374/
https://forum.proxmox.com/threads/multiple-qdevices.98879/

I really see no good reason why somebody would want this in the first place and agree with the consensus in the first thread: If you think that you need two qdevices you should think, whether you need a cluster at all. Migration from one node to the other is still possible (although right now just from the command line, it's expected that next year will see the availability of a GUI option for it) via qm remote-migrate and pct remote-migrate. If you have a situation where you need to shutdown most of the cluster nodes (for maintenance or to save power) your usecases propably doesn't warrant the hassle to deal with high-availability (the usual reason for a cluster). I agree with leestekens take on it:

Why not just do that: work on one node at a time. Then everything will be fine and within the intended and supported design.

What's the problem in first doing maintenance on one node, powering it up and then continue with the next one?

Sounds like less hassle for me than dealing with problems resulting from running a not-supported setup. Especially when we are talking about a homelab.

 

[Daniel_Dog]

It may not be “supported” due to a lack of usage.
But even with a 3 node setup, if possible I would still try to add 2 Qdevives.
This is simply due to the fact that if you take 1 node offline and 1 node just decides to go offline, the hole HA falls apart anyway.

If you take 2 out of 3 nodes offline then, yes no more redundancy.
And if you have 1 out of 3 nodes offline then you still do not have HA if a node fails since it cannot do HA without a over 50% vote.
The only question is if you want to do this automatically (no matter what) or if your fine with changing the voting amount and hope the greater voting host does not go offline while 1 node is offline.

 

[VictorSTS]

The only sensible solution here is to simply disable HA (and wait for CRM and LRM to automatically stop after ~10 minutes) when you have to do maintenance in the cluster, so you don't risk node fencing due to lack of quorum, specially in small clusters. Any other option has drawbacks and it's way more complicated / error prone than simply managing the cluster the way it's meant to be managed.

 

[Red Squirrel]

I'm very new to this myself and just getting into Proxmox so maybe I'm missing something, but can't you just migrate VMs manually off both hosts you want to work on? Of course the other host needs to have enough ram to run all the VMs.

 

[Johannes S]

I'm very new to this myself and just getting into Proxmox so maybe I'm missing something, but can't you just migrate VMs manually off both hosts you want to work on? Of course the other host needs to have enough ram to run all the VMs.

This would work, the trouble starts when you have high-availability but dont want to deal with the implications for your mode of operation. As VictorSTS rightfully said in such cases it's way better to avoid HA. I personally would even go so far and say that you shouldn't run a cluster then. If it's important for you to do maintenance for two nodes at the same time (again: WHY?) you shouldn't run them in a cluster who can't bear the loss of two nodes.

 

[logui]

Thank you all for the valuable feedback, I ended up breaking the 3 nodes cluster, and built a new cluster of 2 nodes with a QDevice on the 3rd host (former node 3) now playing the role of Proxmox Backup Server + NFS Server + QDevice.

Now, I get the flexibility that I can fully turn OFF any of the 2 nodes in the cluster, and the other node is fully functional thanks to the QDevice. I am also hosting some VM's disks on the NFS server that I can quickly migrate across nodes live (all nodes are operating in a 2.5Gpbs ethernet network).

I killed 3 birds (PBS, NFS, QDevice) with the same host, and gained the flexibility to fully shutdown a node for many hours without any Quorum issue, and keeping fully functional the other node.

 

[Johannes S]

I killed 3 birds (PBS, NFS, QDevice) with the same host, and gained the flexibility to fully shutdown a node for many hours without any Quorum issue, and keeping fully functional the other node.

Sounds like a good setup for a homelab, congratulations One thing to consider: Have a plan for an additional backups outside of your NAS+PBS and preferably also outside of your flat/house. The NAS+PBS node since is now a single point of failure, meaning that if it's storage gets broken you loose not only your vm data but also your backups.

 

[logui]

Sounds like a good setup for a homelab, congratulations One thing to consider: Have a plan for an additional backups outside of your NAS+PBS and preferably also outside of your flat/house. The NAS+PBS node since is now a single point of failure, meaning that if it's storage gets broken you loose not only your vm data but also your backups.

Thank you, yes, I know about the single point of failure, I have local backups on each node as well (I do them manually), also planning to add a removable datastore in PBS with an external USB disk, and schedule a second backup job there

 

[UdoB]

I have local backups on each node as well (I do them manually)

One thing I learned decades ago: regularly/frequent backups must get produced automatically. Nobody does a daily/weekly backup manually on the long run...

Just my 2 €¢ - good luck!