[SOLVED] Faster failover - possible?

Razva

Renowned Member
Dec 3, 2013
250
8
83
Romania
cncted.com
Right now it takes about 200 seconds for a VM to be restarted from a failed node. Is there any way to speedup the process? I'm looking for something more like 80-120 seconds, from the point on which the node gets disconnected from the cluster 'til the point on which the VM is fully started on a new node. Is this possible?
 
Fencing need currently ~120 seconds because thats the time where our distributed cluster lock time out.
Basically its, after 60 seconds the outage detection triggers, form now on the current master tries to get the lock.
Worst case he needs another 60 seconds for that (i.e. the lost node renewed its locked directly before it crashed), then recovery may add a few seconds but it should be in the range of 5 to 10 seconds.

The cluster lock timeout period is currently almost hard coded, so you cannot change that easily at the moment - at least not with building your own packages (pve-cluster and pve-ha-manager would be the ones), I'm afraid...
 
Last edited:
I was talking about HA, because - I guess? - that "how content got there" is not influencing the failover system. If I'm wrong please correct me.

Sorry for the confusion ^^ Yes, your assumptions are correct, the (automatic) recovery of VMs/CTs on node failure is HA's work not replication...

But if you use the replication in combination with HA you should know that your only recover from the last replicated state.
Meaning, if you replicate your VM every 15 minutes you could loose up to 15 minutes of storage writes from inside the VM, which may or may not be a problem...
If this VM is just a "Compute Node" this may be OK. But if its, for example, a NFS server then you may get into trouble...
If replication and High Availability is needed then Ceph could be recommended, or glusterFS too.
 
  • Like
Reactions: Razva
Fencing need currently ~120 seconds because thats the time where our distributed cluster lock time out.
Basically its, after 60 seconds the outage detection triggers, form now on the current master tries to get the lock.
Worst case he needs another 60 seconds for that (i.e. the lost node renewed its locked directly before it crashed), then recovery may add a few seconds but it should be in the range of 5 to 10 seconds.

The cluster lock timeout period is currently almost hard coded, so you cannot change that easily at the moment - at least not with building your own packages (pve-cluster and pve-ha-manager would be the ones), I'm afraid...

Hi Thomas -

I know I am reviving a very old thread here. But I am curious if the hard-coded lockout timers are still the same for the current version of PVE.

I am currently testing a new deployment.

1 Cluster
3 Nodes
3 OSD Ceph

After pulling the network connections, it took exactly 120sec for the watchdog to decide that the node was offline. It sent me a couple fencing emails. And then ping on the VM resumed after 110sec. So total just under 4min from the time the cables were pulled.

Does this time match the watchdog timers?
 
I know I am reviving a very old thread here. But I am curious if the hard-coded lockout timers are still the same for the current version of PVE.
Yes.

After pulling the network connections, it took exactly 120sec for the watchdog to decide that the node was offline.
The watchdog only fences the node if it would be powered but not responding (hung up or disconnected from network), it doesn't decide anything. The current HA CRM master node will mark a node as offline after 60s from the last status update of the node, from that time on the CRM will try to get the node lock and, if it could acquire it, recover the services, in the worst case that will start to happen after 120s total (i.e., 60s to start trying to fence plus 60s it takes for the lock to timeout in any case). Service recovery is a fresh start on a new node, if the VM is slow to start that needs naturally adds additional time.

So, after 120s max the VM should be recovered but is only yet starting up, so boot time is on top.
 
Yes.


The watchdog only fences the node if it would be powered but not responding (hung up or disconnected from network), it doesn't decide anything. The current HA CRM master node will mark a node as offline after 60s from the last status update of the node, from that time on the CRM will try to get the node lock and, if it could acquire it, recover the services, in the worst case that will start to happen after 120s total (i.e., 60s to start trying to fence plus 60s it takes for the lock to timeout in any case). Service recovery is a fresh start on a new node, if the VM is slow to start that needs naturally adds additional time.

So, after 120s max the VM should be recovered but is only yet starting up, so boot time is on top.
Thanks for clarifying the process. This helps immensely.
 
in my setup in do have a separate hardware reaction (fencing) implemented that switches of node1 when it fails (or looks like...). Next I would need the command how to the the cluster hat is if definitely "offline", so that the replicas recovery may start on node2. What is the command to immediately start the replication recovery on node2, or the "set offline status" for node1? Thanks already!!!
 
What is the command to immediately start the replication recovery on node2, or the "set offline status" for node1?
Sorry, there's no such command, the cluster lock of the other node always needs to be acquired before.
In theory, you could force the lock release, but that's pretty dangerous stuff and I would not like to post detailed instructions here - you can find out everything from ha/pmxcfs code though.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!