Cluster of 2 not for HA

[toxic]

TLDR: is there a "clean" way to not require quorum or always achieve quorum with a single node?

Clustering and HA is nice but in my homelab setup, there is no way to achieve network filesystem with reasonable performance at achievable price and power usage.
I do have a cluster of 3 nodes with ceph, but I'm more feeling the pressure to have at least 2 up'!n'running than the comfort of being able to break one...

So I'm moving away to a single PvE host on beefier hardware and will only use local-btrfs.

Now the thing is PvE hosts my opnsense router, and it is already setup to failover using carp to a second instance. Now I'm going to setup a second PvE node to run at least this failover instance of opnsense and a few backup services like a reverse proxy and such, on a much smaller node like a N100.

I do not need or want HA on these 2 nodes, but the unified GUI and ability to have the manual "migrate" button would be very nice.

But if I setup a cluster I need a way to tell both of them to not wait for quorum, I mean if I am taking my main R7-node down to upgrade its hardware and still need to reboot the second N100-node, if they are in a cluster the N100-node would refuse to start the opnsense VM until the R7-node is back... That's not good for me, the PvE hosts are in a vlan I don't have access to unless opnsense is running...

I am really not concerned with split brain, both nodes are on the same switch in the same vlan and will never have conflicting VM/CT IDs or run the same CT/VM in HA... So if they don't see each other it's because the other one is down...

I know PvE is targeted at enterprise setups where 3 nodes is not an issue and I understand significant work is going into preventing dumb admins to do crazy things, but I really feel 2 nodes is the way to go for me, if no one has an Idea to have a cluster but disable the quorum properly I will just leave them standalone, use 2 web GUI instead of one, and migrate VM/CTs by backup/restore to/from my NFS server (running inside a privileged LXC on my main R7-node...)

I read about "pvecm e 1" but it's unclear to me, seems it can't be done while 2 nodes are up as there are 2 votes already, and can only be done on one node while it's the last remaining live node, so that would mean manual intervention, and even that it seems the command might not work for some users...

Any advice is welcome, just saying I don't believe one can convince me to raise the count to 3 machines, even a raspberry q-device, I really don't see how adding a raspberry that I will fail to upgrade regularly would help my setup besides making PvE believe I have a bigger cluster than I really do

Thanks in advance

 

[esi_y]

TLDR: is there a "clean" way to not require quorum or always achieve quorum with a single node?

Clustering and HA is nice but in my homelab setup, there is no way to achieve network filesystem with reasonable performance at achievable price and power usage.
I do have a cluster of 3 nodes with ceph, but I'm more feeling the pressure to have at least 2 up'!n'running than the comfort of being able to break one...

So I'm moving away to a single PvE host on beefier hardware and will only use local-btrfs.

Now the thing is PvE hosts my opnsense router, and it is already setup to failover using carp to a second instance. Now I'm going to setup a second PvE node to run at least this failover instance of opnsense and a few backup services like a reverse proxy and such, on a much smaller node like a N100.

I do not need or want HA on these 2 nodes, but the unified GUI and ability to have the manual "migrate" button would be very nice.

But if I setup a cluster I need a way to tell both of them to not wait for quorum, I mean if I am taking my main R7-node down to upgrade its hardware and still need to reboot the second N100-node, if they are in a cluster the N100-node would refuse to start the opnsense VM until the R7-node is back... That's not good for me, the PvE hosts are in a vlan I don't have access to unless opnsense is running...

I am really not concerned with split brain, both nodes are on the same switch in the same vlan and will never have conflicting VM/CT IDs or run the same CT/VM in HA... So if they don't see each other it's because the other one is down...

I know PvE is targeted at enterprise setups where 3 nodes is not an issue and I understand significant work is going into preventing dumb admins to do crazy things, but I really feel 2 nodes is the way to go for me, if no one has an Idea to have a cluster but disable the quorum properly I will just leave them standalone, use 2 web GUI instead of one, and migrate VM/CTs by backup/restore to/from my NFS server (running inside a privileged LXC on my main R7-node...)

I read about "pvecm e 1" but it's unclear to me, seems it can't be done while 2 nodes are up as there are 2 votes already, and can only be done on one node while it's the last remaining live node, so that would mean manual intervention, and even that it seems the command might not work for some users...

Any advice is welcome, just saying I don't believe one can convince me to raise the count to 3 machines, even a raspberry q-device, I really don't see how adding a raspberry that I will fail to upgrade regularly would help my setup besides making PvE believe I have a bigger cluster than I really do

Thanks in advance

TLDR answer is to use a QDevice.

Alternatively twonode option but (!) understand what wait_for_all means in that scenario [1]. Or use auto_tie_breaker. Then there are hacks like deciding in which node is the "master" and let it have it 2 votes (with the consequence that one node going down loses the quorum).

NB Losing quorum is not any detrimental to your running VMs on the inquorate node anyhow.

Just, do not use HA, is probably the answer you are looking for.

[1] https://manpages.debian.org/unstable/corosync/votequorum.5.en.html

 

[toxic]

My understanding was that a node will not start any VM unless quorum is achieved, in my case the impact is that from cold start, if I start only one of the 2 nodes it will not start the VM/CT that are set to start on boot as it doesn't have quorum.
I will test that anyway I now have an unused 3 node former cluster to test things out

I believe two_node 1 in conjunction with wait_for_all 0 is what I need to have the cluster GUI, the ability to migrate a VM with one click (needs to be stopped/migrate/start I guess between my amd and Intel node anyway), and no single node would wait for the other. Now I'll have to be careful anyway manipulating VMs while one node is down I can understand I need to avoid conflicts in the /etc/pve content...

Thanks a lot for the quick response!

Will consider the q-device once I re-do my networking and can put it on my Synology NAS but for now it's not in the same VLan and that wouldn't be helpful with my router being virtualized...

 

[esi_y]

My understanding was that a node will not start any VM unless quorum is achieved, in my case the impact is that from cold start, if I start only one of the 2 nodes it will not start the VM/CT that are set to start on boot as it doesn't have quorum.

This is correct, but why is this a concern? Is it common case that one of your nodes will be mostly not running? As you discovered, for manual interventions, you can override that with pvecm expected 1. This does nothing else than corosync-quorumtool -e 1 which you can check out for yourself [1] basically manipulates the votequorum option for you. With two_node: 1, you already get that from the start though.

EDIT: See [2] again:

two_node: 1

Enabling two_node: 1, quorum is set artificially to 1.

NOTES: enabling two_node: 1 automatically enables wait_for_all.

wait_for_all: 1

When WFA is enabled, the cluster will be quorate for the first time only after all nodes have been visible at least once at the same time.

It will test that anyway I now have an unused 3 node former cluster to test things out

I believe two_node 1 in conjunction with wait_for_all 0 is what I need to have the cluster GUI

It will work, but this means in the worst case scenario both nodes can boot up at once with no network with each other and simply start creating VMs with same IDs, etc. I know it's a homelab, but what's the benefit of this in the light of the above? The reason it defaults to wait_for_all: 1 is to know that the network is (or at least once was) alright and if the other node is gone, it is safe to assume it is off.

, the ability to migrate a VM with one click (needs to be stopped/migrate/start I guess between my amd and Intel node anyway), and no single node would wait for the other. Now I'll have to be careful anyway manipulating VMs while one node is down I can understand I need to avoid conflicts in the /etc/pve content...

Yes.

Thanks a lot for the quick response!

Will consider the q-device once I re-do my networking and can put it on my Synology NAS but for now it's not in the same VLan and that wouldn't be helpful with my router being virtualized...

QDevice can be on the other side of the planet. It's a TCP connection no latency requirements whatsover. When it's down, no adverse effect (as opposed to having just a 2-node cluster).

[1] https://manpages.debian.org/unstable/corosync/corosync-quorumtool.8.en.html
[2] https://manpages.debian.org/unstable/corosync/votequorum.5.en.html

 

[toxic]

I expect to leave the second node off for the majority of the time, it's going to be powered on when I'm away (home assistant will schedule that) to keep a remote access in case something breaks. It already happened, my PSU of course waited for me to be away on holiday to fail last year...Cheap Chinese bricks...
The switch is enterprise grade even if old (juniper ex3300) so I'm not concerned with networking going bad unless I break something and all nodes are plugged directly into it.
But if I put the q-device in another vlan it will vote only once my (virtual) router is up and can forward packets between vlans, so that won't fly, but I'll dig into vlans on my Synology I know it's feasible, and I get it would make sense.

But as you guessed I'm not worried to create or start VMs with the same ID, I'm much more worried to have my virtual router not started because of quorum issues I already have a third failover that I was hoping to get rid of because power price in Europe....

 

[esi_y]

I'm much more worried to have my virtual router not started because of quorum issues I already have a third failover that I was hoping to get rid of because power price in Europe....

I know this is not what you were asking in this thread, but why does the PVE host need to route its own traffic through the virtual router?

 

[toxic]

PvE host is on my servers VLAN separate from my Clients VLAN where my NAS lives because historical reasons and poor discovery on windows/samba part.
So servers don't need a router to work together, but to get to internet or to provide services to clients then yes router is needed and the router is virtualized.
Partly because managing fw rules in opnsense is way nicer but mostly because I don't want to manage 2 firewalls one in opnsense and one I would need to learn on the proxmox host itself.
Physically nothing stops it from accessing all vlans but only the servers VLAN is known to pve

 

[exp]

@toxic i have the same scenario as you (even with the virtualized router).

Which solution did you end up doing?