cluster and VM boot order - race conditions - best practice?

[sigmarb]

Hi folks,

we have several multi-node proxmox clusters. Sometimes we have to cold-start all nodes (full power outtake onsite) that causes a race-condition between VMs.

Servers start before firewall/domain controllers are up. (wrong network profiles on windows)
Application servers start before sql-server is up and so on (manual intervention required).

I'm aware that one can not set boot order in case of cluster management is on. How do you handle this situation?

I was thinking about simply keep all machines in state 'ignored' in HA/resources menu and start them in pre-defined order by skript.
However this causes problems when one node goes down as the cluster does not feel responsible for the VMs anymore.

Ideas/thoughts?

We do not have the manpower to have 24/7 staff avail to start VMs manually in order after power outtake.

Thank you.

 

[tim]

What about VM->Options->Start/Shutdown Order?

 

[sigmarb]

This is ignored if HA is active. See your own admin manual

 

[tim]

You're right, did get the meaning wrong here thought you were thinking about using HA, not already using it.

In general this looks like an orchestration problem, an easy solution would be to restart the service/vm which isn't running properly.
Let HA handle the start part and just use your script to shutdown guests with services reporting problems.

Even if there would be an option, how would it look like?
I mean if something happens on boot or the service isn't starting fast enough, still an issue.

 

[sigmarb]

Well your cluster resource manager should handle this. For example, with pacemaker cluster you simply set constraints between the resources:

https://access.redhat.com/documenta..._add-on_reference/ch-resourceconstraints-haar

This could be used to honor the pre-defined boot orders of VMs in proxmox GUI you outlined before.