Need Clarification on High Availability / CRS Behavior

[sk.]

Hello,

I'm a bit confused about High Availability and the Cluster Resource Scheduler (CRS). I don't think I've fully understood how High Availability works yet. Could you help me understand it better?

I'm referring to the following scenario.

Let's assume the cluster has five nodes, and all of them are running at approximately 60% utilization. If one node fails, all HA virtual machines on that node are stopped and then restarted on the remaining nodes.

My concern is this: when one node fails in a five-node cluster, if all of its HA VMs are restarted on a single one of the remaining four nodes rather than being spread across them, that node will have %60 + %60 load and it could become overloaded and fail as well, creating a cascading failure loop.

If I understand it correctly, when rebalance-on-start is working as intended, it evaluates the available capacity while restarting the HA resources after a node failure and starts each resource on the most appropriate node with sufficient available load.

However, since this feature is still in Technology Preview, my understanding is that if it does not trigger or does not work as expected, there is still a possibility of entering a node failure loop. Am I thinking about this incorrectly?

Even when the scheduling mode is set to dynamic load, I'm not confident that, in a scenario where rebalance-on-start is not functioning correctly, all HA resources from the failed node will be distributed properly across the remaining four nodes.

My understanding is that automatic rebalance is different from rebalance-on-start. Automatic rebalance does not distribute HA resources while they are being restarted after a failure. Therefore, if rebalance-on-start does not take effect, all HA resources could initially be started on a single node. Only after they are already running would automatic rebalance migrate them according to load. By that point, however, if too many HA resources are powered on simultaneously, the node could become overloaded very quickly. If that overloaded node also fails, the cascading failure loop would be unavoidable.

 

[dakralex]

Hi!

My concern is this: when one node fails in a five-node cluster, if all of its HA VMs are restarted on a single one of the remaining four nodes rather than being spread across them, that node will have %60 + %60 load and it could become overloaded and fail as well, creating a cascading failure loop.

In case of a fenced node, the HA resources on the fenced node are put in recovery state and the HA Manager will select a recovery node for each HA resource individually. This will always try to spread the HA resources on the remaining cluster nodes with respect to the HA affinity rules of the to-be-recovered HA resources.

The Scheduler Mode (either basic, static-load, or dynamic-load) [0] determines according to which data the scheduling decisions are made, which are mainly the recovery node selection, the rebalance-on-start selection, and the load balancer.

My understanding is that automatic rebalance is different from rebalance-on-start. Automatic rebalance does not distribute HA resources while they are being restarted after a failure. Therefore, if rebalance-on-start does not take effect, all HA resources could initially be started on a single node. Only after they are already running would automatic rebalance migrate them according to load. By that point, however, if too many HA resources are powered on simultaneously, the node could become overloaded very quickly. If that overloaded node also fails, the cascading failure loop would be unavoidable.

The recovery node selection, the rebalance-on-start selection and the automatic rebalance decisions are entirely different from each other. The recovery selection will always try to spread the to-be-recovered HA resources among the remaining nodes. The rebalance-on-start feature does only work on requested-to-start HA resources. The automatic rebalancing decisions are only made for already running HA resources.

The rebalance-on-start feature does try to prevent that starting many HA resources at once will crash a node because it cannot host all of them by moving them away if there's a better fitting node and balance the load across the cluster at the HA resource startup.

[0] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#_crs_scheduling_mode

 

[sk.]

The recovery selection will always try to spread the to-be-recovered HA resources among the remaining nodes. The rebalance-on-start feature does only work on requested-to-start HA resources. The automatic rebalancing decisions are only made for already running HA resources.

The rebalance-on-start feature does try to prevent that starting many HA resources at once will crash a node because it cannot host all of them by moving them away if there's a better fitting node and balance the load across the cluster at the HA resource startup.

[0] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#_crs_scheduling_mode

Hi @dakralex,

As I understand it, the rebalance-on-start feature does not take any action for VMs that are being starting on another node after a node failure. It only takes effect when a start is issued for VMs that are already in a stopped state. After a node failure, the HA Manager decides which node the VM will be restarted on, based on the selected scheduling mode. Is that correct?


If that's the case, then my understanding is that when the scheduling mode is set to dynamic-load, the HA Manager would never restart all VMs on a single host that is already heavily loaded, thereby avoiding a fail loop. Could you please confirm whether my understanding is correct?

 

[dakralex]

As I understand it, the rebalance-on-start feature does not take any action for VMs that are being starting on another node after a node failure. It only takes effect when a start is issued for VMs that are already in a stopped state. After a node failure, the HA Manager decides which node the VM will be restarted on, based on the selected scheduling mode. Is that correct?

Yes, to be exact rebalance-on-start will only be relevant if the HA resource goes through the request_start state, which can only happen if the HA resource was previously in the stopped state and e.g. a user requests the guest to be started through the web interface, or if the HA resource is added to the HA stack without specifying any state (and started is the default there).

In case of a node failure, HA resources will first be in fence state to wait whether the node gets fully fenced, then go in the recovery state to select a recovery node, then be moved to that node, and at last be put in started state again.

If that's the case, then my understanding is that when the scheduling mode is set to dynamic-load, the HA Manager would never restart all VMs on a single host that is already heavily loaded, thereby avoiding a fail loop. Could you please confirm whether my understanding is correct?

The scheduling mode does only specify which data is used to decide that process: (1) basic: use the count of running guests on each node, (2) static-load: use the total amount of configured CPU and memory resources for the guests on each node, or (3) dynamic-load: use the currently used amount of CPU and memory resources on each node.

The HA Manager will spread the HA resources out on the cluster nodes regardless of the scheduling mode.