HA Failover Logic

[M@ario]

Hello!

Can anybody tell me how proxmox distributes VMs in case of a failed cluster node?
For example if I have a 3 node cluster (Node1-3). Node1 and 2 have several VMs (RAM is nearly full), Node 3 is empty. If Node1 fails, will proxmox relocate the failed VMs on Node2 (probably extending physical RAM) or will it be clever and relocate them to Node3?

 

[dietmar]

you can define failover domains - see 'man rgmanager'

 

[M@ario]

OK. Failover domains would work for my special example. But they are static and keeping them up-to-date seems hard in a real enviroment, where CPU and memory consumption change all the time. The only possibility I see is to always have an complete empty spare node, like in my example.
Are there any plans to replace rgmanager with pacemaker, which seems to be able to relocate resources based on available system resouces (http://theclusterguy.clusterlabs.org/post/570381880/feature-spotlight-utilization)?

 

[dietmar]

Are there any plans to replace rgmanager with pacemaker

No. But IMHO dynamic resource relocation is a dangerous feature anyways.

 

[hansm]

No. But IMHO dynamic resource relocation is a dangerous feature anyways.

It depends. IMHO there shouldn't be relocations based on available resources during the day. This should only occur when a node crashes and the VM's that were running on that node are automatically migrated to other nodes. The current situation is unpredictable, or maybe it's like node1 migrates to node2, node2 to node3, etc. But that's far more dangerous than migrating VM's based on available resources on other nodes. I think when HA comes in action it should make the decision on where to place a VM based on available resources. ATM it could happen on a 3-node cluster with 64GB RAM per node and 40GB RAM per node assigned to VM's that 80GB RAM VM's are placed on 1 node with only 64GB RAM available. If this is based on available resources there shouldn't be a problem because both remaining nodes are getting VM's with a total of 60GB RAM per node.

When the failed node comes back online it should be a manual action to migrate VM's back to it.

This is my most wanted feature

 

[dietmar]

This is my most wanted feature

We plan to use our own HA manager in PVE 4.0, which replaces rgmanager. The code is already online:

https://git.proxmox.com/?p=pve-ha-manager.git;a=summary

And that code implement such logic, although it just distribute VM evenly among all available nodes.

 

[hansm]

Wow! This is great. Thank you!

By distributing evenly among all available nodes, you mean that in a 3-node cluster where node1 with 10 VM's fails, there will go 5 VMs to node2 and 5 VMs to node3? This is far better than the current method, but if you take into account that nodes in a cluster can have different hardware specs it isn't failsafe IMHO. In the Proxmox VE UI you can already see the memory total and used, and CPU utilization, can't that be used to make distribution decisions?

Need to say that I'm really very happy with the new HA manager planned for PVE 4.0! Is there already an ETA for 4.0?

 

[dietmar]

By distributing evenly among all available nodes, you mean that in a 3-node cluster where node1 with 10 VM's fails, there will go 5 VMs to node2 and 5 VMs to node3?

yes

This is far better than the current method, but if you take into account that nodes in a cluster can have different hardware specs it isn't failsafe IMHO. In the Proxmox VE UI you can already see the memory total and used, and CPU utilization, can't that be used to make distribution decisions?

Yes, but somebody need to implement it ;-)