2 node cluster - adding a qdevice or an additional node

[mgaudette112]

Hi,

I have a 2 nodes cluster running - each node can easily run the entirety of the existing VMs if needed, so I really don't need a third node at the moment. I already have a separate generic device setup for quorum.

I am in the process of setting up a PBS on an available server. I would like to know what would be the recommended way of setting up my system when it comes to cluster and qorum:

1) use the PBS server as the qdevice? (removing qdevice from the existing generic device)
2) Instead of a pure PBS, setup a PVE + PBS (both on the bare metal device) on that third node, turning it into a third PVE node for the purpose of clustering, even if no VMs will even run on it.
3) Keep on using my other qdevice and not setup anything qdevice-related on the PBS

I am leaning towards 1 or 2, as option 3 forces me to take a 4th device into consideration for PVE-related purposes - I do have appropriate devices for this purpose, but I'd rather avoid adding a 4th pieces to this puzzle.

Option 1 looks like the cleanest option, but I'm wondering if there is any benefit to having a real PVE as the third device.

 

[leesteken]

PBS runs fine (for homelab purposes) as a container in PVE and can even snapshot backup itself (although I would backup the actual storage). I would think that 3 PVE nodes would give you the most flexibility (if you intend to run all three all the time).

 

[mgaudette112]

PBS runs fine (for homelab purposes) as a container in PVE and can even snapshot backup itself (although I would backup the actual storage). I would think that 3 PVE nodes would give you the most flexibility (if you intend to run all three all the time).

I've seen the PBS in a VM option and the PBS/PVE both running on the same bare metal option. Given that I am using ZFS (on my PVEs and PBS) it seems better to run both on bare metal as opposed to PBS in a VM. I have more ZFS general knowledge than Proxmox-specific knowledge, I must admit.

 

[leesteken]

I've seen the PBS in a VM option and the PBS/PVE both running on the same bare metal option. Given that I am using ZFS (on my PVEs and PBS) it seems better to run both on bare metal as opposed to PBS in a VM. I have more ZFS general knowledge than Proxmox-specific knowledge, I must admit.

Maybe it's just a mix-up in terminology but I don't recommend running PBS in a VM (with ZFS on top of ZFS). I think running PBS in a container is almost as good as bare-metal as it can directly use (the same) ZFS storage (since it is not a VM), while still being on a separate life cycle (except for the kernel) and easily backed up (and moved to another node, in case of shared storage).

 

[esi_y]

Hi,

I have a 2 nodes cluster running - each node can easily run the entirety of the existing VMs if needed, so I really don't need a third node at the moment. I already have a separate generic device setup for quorum.

Not asked here: Do you prefer to repurpose the "generic device" for something else than QD?

I am in the process of setting up a PBS on an available server. I would like to know what would be the recommended way of setting up my system when it comes to cluster and qorum:

1) use the PBS server as the qdevice? (removing qdevice from the existing generic device)

If had to go get an extra device for QD, I would suggest this is just fine.

2) Instead of a pure PBS, setup a PVE + PBS (both on the bare metal device) on that third node, turning it into a third PVE node for the purpose of clustering, even if no VMs will even run on it.

I would definitely not want to have PVE + PBS together on OS, especially I have no use for that PVE 3rd node from what you mentioned. Also in certain aspects QD when setup outside the network, etc. has benefits over just a regular node's vote.

3) Keep on using my other qdevice and not setup anything qdevice-related on the PBS

I am leaning towards 1 or 2, as option 3 forces me to take a 4th device into consideration for PVE-related purposes - I do have appropriate devices for this purpose, but I'd rather avoid adding a 4th pieces to this puzzle.

Fewer pieces of hardware also mean that e.g. should you need to go about reinstalling the PBS which doubles as QD, you are left without QD for the time being.

Option 1 looks like the cleanest option, but I'm wondering if there is any benefit to having a real PVE as the third device.

See my initial question.

I've seen the PBS in a VM option and the PBS/PVE both running on the same bare metal option. Given that I am using ZFS (on my PVEs and PBS) it seems better to run both on bare metal as opposed to PBS in a VM. I have more ZFS general knowledge than Proxmox-specific knowledge, I must admit.

I also find running PBS as a VM very strange setup. For recovery you need then that one node to be available or recover the VM first from somewhere else ... and data stored off cluster.

 

[mgaudette112]

Not asked here: Do you prefer to repurpose the "generic device" for something else than QD?

Right now that device is used for something and runs qdevice also for the PVE cluster. But the plan is to decommission it completely eventually, at which point I need another device to take over the qdevice service. At this point it feels like it will be a pure PBS, with a qdevice service.

 

[mgaudette112]

Fewer pieces of hardware also mean that e.g. should you need to go about reinstalling the PBS which doubles as QD, you are left without QD for the time being.

Making my cluster go from the ideal 3 nodes/qdevice to 2 - but it would still run properly right, as long as both PVE nodes function properly?

Couldn't the same thing be said about any single PVEs ? If one needs maintenance, during this window the system is left with only 2 functioning voters.

 

[esi_y]

Making my cluster go from the ideal 3 nodes/qdevice to 2 - but it would still run properly right, as long as both PVE nodes function properly?

It will. But e.g. if one has ARM device running just QD, it's very unlikely that one ever needs going down for anything, as opposed to having it stuffed somewhere alongside other services.

Couldn't the same thing be said about any single PVEs ? If one needs maintenance, during this window the system is left with only 2 functioning voters.

There's other setups possible, e.g. no QD, but give one of the two nodes 2 votes. That way, you have a cluster with a "master" node which cannot go down, but the other node can go down. It does not require a QD, it is also not as good as QD, but it can sustain 1 ("slave") node down as opposed to no nodes down (no QD, equal votes setup).

 

[mgaudette112]

It will. But e.g. if one has ARM device running just QD, it's very unlikely that one ever needs going down for anything, as opposed to having it stuffed somewhere alongside other services.

Well, I do have a (planned) PBS I can use, which is drawing power, taking space anyways. I consider my 2 PVEs and PBS as important enough that they will not be down, planned maintenance aside (firmware, updates, etc.). And if they suffer an unplanned issue, the priority would be to bring what's down back to life. I do have cheap ARM devices lying around through, but they also are less naturally reliable (not running redundant drives or PSUs). Any downside in running two qdevice (in addition to the two PVE nodes), the PBS and a cheap ARM device?

Can we ever have too many qdevices tacked on to a PVE cluster, if the hardware is already powered on anyways?

There's other setups possible, e.g. no QD, but give one of the two nodes 2 votes. That way, you have a cluster with a "master" node which cannot go down, but the other node can go down. It does not require a QD, it is also not as good as QD, but it can sustain 1 ("slave") node down as opposed to no nodes down (no QD, equal votes setup).

That doesn't sound good - the master node will eventually need maintenance (again, firmware, OS updates, etc.) bringing down the cluster's functionality. The point of having each PVE strong enough to run all VMs is that maintenance can be done on either PVEs without any consequences on the VMs (as long as the VMs are moved prior or setup as HA).

 

[esi_y]

Well, I do have a (planned) PBS I can use, which is drawing power, taking space anyways. I consider my 2 PVEs and PBS as important enough that they will not be down, planned maintenance aside (firmware, updates, etc.). And if they suffer an unplanned issue, the priority would be to bring what's down back to life. I do have cheap ARM devices lying around through, but they also are less naturally reliable (not running redundant drives or PSUs). Any downside in running two qdevice (in addition to the two PVE nodes), the PBS and a cheap ARM device?

Can we ever have too many qdevices tacked on to a PVE cluster, if the hardware is already powered on anyways?

You can't have more than one QD at a time. A single QD can act as such to multiple clusters, but a cluster cannot have multiple QDs "enrolled".

That doesn't sound good - the master node will eventually need maintenance (again, firmware, OS updates, etc.) bringing down the cluster's functionality.

I am not endorsing it, but for two-node setups it is possible to get creative. For planned maintenance, you can set pvecm expected 1 prior to taking the 2-vote "master" down.

The point of having each PVE strong enough to run all VMs is that maintenance can be done on either PVEs without any consequences on the VMs (as long as the VMs are moved prior or setup as HA).

It's your call, the QD can also run somewhere completely off, e.g. VM. Most people have something "satellite" running already. Unlike the requirement on the nodes networking, QD may be quite a deal of latency away.

 

[mgaudette112]

Unlike the requirement on the nodes networking, QD may be quite a deal of latency away.

I did not realize that. Thank you for all this information, it does clarify quite a bit of things.