How to achieve both HA and Autostart combined with proper shutdown on power outage?

[Wazaari]

Hi all,

our setup is a three node Proxmox Cluster with Ceph Storage. So far, everything works fine. We're organised VMs in three pools (Tier2, least important, Tier1, medium important, Tier0, critical). Autostart is configured for all of them, HA is configured for the Tier0 VMs only to make sure they survive a host failure.

In addition, the hosts are connected to a UPS system mainly to ensure proper shutdown in case of power outage. The UPS system is monitored, and the following things happen:

- At power loss (battery event), all Tier2 VMs are shut down
- At 60% capacity, all Tier1 VMs are shut down, then the third node is shut down as well
- At 20% capacity, all Tier0 VMs are shut down, then the two remaining nodes are shut down

This works fine, but after restarting everything, the Tier0 VMs are not started automatically. My assumption is that this has something to do with the HA state of these VMs, as the HA state first goes into `request_stop` and then to `stopped` during the shutdown event.

So the question is: how can I properly shut down HA protected VMs during the power outage, while still making sure they get started again once power returns?

Thanks!
Daniel

 

[hec]

I think your problem is that you stop 3rd node first.

I would stop all 3 nodes after all vms are in stopped state.

 

[Wazaari]

Thanks for your answer, I'm not sure I'm understanding it though. Can you elaborate why that would make a difference and how it would affect the startup order?

 

[hec]

Can you post your startup order?

 

[Wazaari]

Hi, sure. As startup is handled on a per host basis, I'll give them as per-host as well:

Proxmox Node 01

VM Name	Pool	Order	Startup Delay
firewall	Tier0	10	N/A
opvpn-ra	Tier0	10	N/A
k8s-node2	Tier1	20	30s
packetfence	Tier1	30	30s

Proxmox Node 02

VM Name	Pool	Order	Startup Delay
dns	Tier0	10	N/A
dc1	Tier0	10	N/A
k8s-node2	Tier1	20	30s
homeassistant	Tier2	30	30s
smbv1_proxy	Tier2	40	30s

Proxmox Node 03

VM Name	Pool	Order	Startup Delay
k8s-node3	Tier1	20	30s
cups	Tier2	30	30s
ima_winxp	Tier2	40	30s

Node03 doesn't contain any Tier0 VMs, that's also why we shut it down in case of power outage. After restore, all VMs come back up, except for the three Tier0 VMs. Those happen to be the HA enabled ones, too, so I suspect there's some correlation as described initially.

Note that I also have a generic 30s startup delay configured on all hosts, to make sure the Ceph FS is ready before VMs are started,

 

[hec]

Do you start all 3 nodes first and then the VMs?

 

[Wazaari]

I'm sorry, what do you mean? I start all three nodes in parallel (at the same time), I'm not doing any manual actions on the VMs.

 

[hec]

Ok this should be fine

Can you try to stop the 3rd node after all VMs are in stopped state?

 

[LordRatner]

From the documentation:

VMs managed by the HA stack do not follow the start on boot and boot order options currently. Those VMs will be skipped by the startup and shutdown algorithm as the HA manager itself ensures that VMs get started and stopped.

https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_bootorder

You could skip the step where you shutdown the Tier 0 guests and just shutdown the nodes, which will automatically stop all the guests. They will restart when the node starts.

 

[Wazaari]

Thanks @LordRatner, somehow managed to miss this piece of documentation. Guess thats the only way then. I was worried about timing effects, as the two hosts would also be the last two CEPH members. If one of the hosts goes down before all guests shut down from the other host, the guests wouldn't be able to write to disk anymore.

 

[dooferorg]

I happened across this thread (apologies if it's a bit old) since I was also wondering what I was missing regarding the automatic start of high-availability entries after the systems were gracefully shut down. After deciding that all I really needed to do was to prod the HA manager, I made myself a simple BASH script to do what I needed.

How I use it is to run it on a node once the cluster is operational. You could in theory plop it in a cron job or /etc/rc.local but I want to be sure things were kind of operational before I spin everything up.

I thought I'd share it here in case anyone finds it useful!

#!/bin/bash

HASELECTION=`ha-manager status | grep service | grep stopped | awk {'print $2'} | awk -F ':' {'print $2'}`
for IDNUM in $HASELECTION
do
  echo $IDNUM
  ha-manager set $IDNUM --state=started
done