Proxmox 3 node cluster - recover with 1 node after power outage

[mati]

Hi folks,

Background:
I'm running a Proxmox Cluster with 3 nodes at home, where 1 node has OPNsense VM and Omada Controller in LXC:

• pve0 (OPNsense + Omada)
• pve1
• pve2

I have installed NUT (Network UPS tools) that will turn off pve1 and pve2 when UPS is on a battery for more than 5 minutes. pve0 and my network gear will keep running until UPS battery dies and it will boot up once power is restored.

I don't use any HA stuff and nodes operate independently (except pve0 being a router). I'd prefer to keep them as a cluster as I really like having the ability to move a VM between nodes and having 1 UI to manage all my VMs/LXCs.

Issue: When the power is restored VMs on pve0 won't boot as there is no quorum.

Potential solutions:
I thought about the following options

1. Set pve1 and pve2 nodes in BIOS to boot when the power is restored:
   • pve1 has TrueNAS Scale VM with a few HDDs and when the power is restored, it could be restored for only a few seconds/minutes and I'd really like to avoid
2. Maybe connect pve1 and pve2 through some smart plugs and have pve0 monitor these plugs and only turn the power on if they're on for more than X minutes. Although I'm not sure it is feasible, also it gets a bit complex and adds multiple points of failure (plugs + automation).
3. Similar to option 2, but set up Wake-On LAN on pve1 and pve2 as Raspberry PI or other device power directly via UPS USB port that will only power on these 2 nodes when UPS is on line power and has at least 50% of the battery. Not sure if Wake-On LAN is a good option for Proxmox with TrueNAS Scale VM.
4. Increase voting power from 1 to 2 for pve0
   • Seems like the easiest solution, but I know it's discouraged. I don't fully understand what are the potential bad consequences of this setup in my case.
   • If pve0 goes down, I won't be able to start/stop VMs from other nodes, but at this point I will have bigger problems as my router will be down. Worst case, if I need to change something on pve1 and pve2 when pve0 is down, I could simply disconnect pve1/pve2 from the switch, run pvecm expected 1 and do whatever I want on the node.

Do you have any recommendations? Are these feasible options? What potential issues I am missing in case 4?

I'm also open to different solutions to solving this problem. Power outages are not frequent, but I need to be able to restore connectivity without human intervention.

Thanks so much and thank you for building an amazing product!

 

[sw-omit]

For option 1: A lot of BIOS/ILO/IPMI's have an option to wait X seconds before booting, and on top of that you can also set an auto-start delay for VM's on proxmox itself, both would help to limit any potential issues on the VM-side (still might be bad for proxmox itself though).
For option 2: Indeed only adds complexity / point-of-failures I would say. What might be an option though is that on some UPS-models, you can define groups (aka the left or right set of power-outputs) and turn them off/on individually via API-calls, I would have more trust in that then smart-plugs personally.
For option 3: Not sure why Wake-On-Lan would not be a good idea for ProxMox, it's basically just an automatic way for someone to press the power-button on the servers.
For option 4, you'd have to change the voting power to 3, since then it would have 3/5 votes instead of 2/4 (which is exactly 50% and quorum needs more then 50%)


Also, from what I vaguely recall, indeed if there is no quorum on boot auto-start does not work, BUT once quorum is established (so once one of the other nodes has started up) the auto-start will be triggered then still, maybe with the added auto-start delay option to give 2 of the servers the chance to get into quorum before kicking the auto-start off.

 

[mati]

For option 1: A lot of BIOS/ILO/IPMI's have an option to wait X seconds before booting

Oh, I didn't know that, thanks for the tip!

some UPS-models, you can define groups (aka the left or right set of power-outputs) and turn them off/on individually via API-calls,

I would need to double check that, but I'm 95% sure that in my consumer grade equipment don't have that option

Not sure why Wake-On-Lan would not be a good idea for ProxMox

TBH I had no idea how it works until a few hours before that post and read just a bit about these different sleep states (S3, S4) and I wanted to check if it's ok to do it when running a NAS (IIRC one should not put a NAS to sleep). Also wasn't sure if Wake-on-LAN will work well if UPS fully discharges and the devices are disconnected from power.

For option 4, you'd have to change the voting power to 3, since then it would have 3/5 votes instead of 2/4 (which is exactly 50% and quorum needs more then 50%)

Thanks, any caveats to be aware of if I go with that option?

Also, from what I vaguely recall, indeed if there is no quorum on boot auto-start does not work, BUT once quorum is established (so once one of the other nodes has started up) the auto-start will be triggered then still, maybe with the added auto-start delay option to give 2 of the servers the chance to get into quorum before kicking the auto-start off.

I will spent a bit more time thinking about this, as I have 2 separate UPSes (1 for servers and 1 for network gear), so I can power down the servers and run another one until battery dies and still use the Internet.

maybe with the added auto-start delay option

I also noticed that Omada AP when not connected to the controller and Internet at some point after the boot will stop broadcasting SSID (not sure why though). Need to have a closer look, to make sure the connectivity can be self-restored after the outage.

Thanks so much for the help!

 

[Heracles31]

If you give 3 vote to pve0, pve0 can never goes down without breaking quorum and never will pve1 and 2 be able to achieve quorum by themselves because they total 2 votes out of 5…

 

[sw-omit]

I know that for Unifi AP's there is an "uplink monitor" option in the setting which, if it notices there is no more uplink, stop broadcasting so that devices are able to roam to other AP's that might have internet still.

As for the other option, I would generally advice to not "mess" with the quorum-settings and keep to one of two supported setups, being odd amount of nodes all with one vote, or even amount of nodes and an added QDevices.

One option you COULD go for (not fully supported but would give a bit flexibility) would be to give pve0 2 votes and pve1/2 1 vote, AND then add a QDevice on something like a Raspberry-Pi.
That way pve0+Q or pve1+2+Q would both be in quorum, but does mean that if Q and ANY of the nodes goes down (even if it is Q+2, so you still have 0+1 for 3 votes) it loses quorum because a Q-Device is set