About CRS Dynamic-load Scheduler Status

O

oktay454

Member
May 7, 2022
5
0
21
31
Hello,

Up until now, for every technology preview introduced to Proxmox, I used to see a warning along the lines of "this feature is a technology preview" directly in the respective user interface.
However, for the dynamic load scheduler feature added to CRS, there is no such warning in the interface, even though the manual explicitly states it is a technology preview:
For the static-load and dynamic-load scheduler modes, thisfunctionality is still in technology preview. The more HA resources the morepossible combinations there are, so it’s currently not recommended to use it ifyou have thousands of HA resources.
Click to expand...
Could you please clarify the current status of this feature?

https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#_ha_resource_rebalance_on_start_opt_in
 
Hi!

The section you're referring to states that the rebalance on start feature is in technology preview, not the CRS load balancer. The technology preview status for the CRS load balancer has been removed recently. But you are right, that the rebalance on start option is lacking warning in the web UI. Thanks for pointing that out!

Update: also sent a patch. Thanks @oktay454 !
 
Last edited:
  • Like
Reactions: dakralex and oktay454
@mkoeppl, I have one additional question: when one of the nodes goes down, HA kicks in and redistributes the VMs from the failed machine. Does the CRS load balancer come into play at this stage? In other words, does it distribute the resources evenly?
 
  • Like
Reactions: oktay454
Right now, we're concerned that when we enable HA in our existing cluster, if a node fails, the VMs may end up being concentrated on a single node or drive one of the remaining nodes to 100% resource utilization. If that happens, the overloaded node could also become unresponsive, potentially causing a failover loop where the node taking over the HA resources also fails. As a result, the entire cluster could go down.

As I understand it, preventing this scenario requires the "rebalance on start" feature to move out of Technology Preview.

Given the current state of the feature, what would be the best configuration or best practice to minimize this risk?

1785318558862.png

With the configuration shown in this screenshot, is it guaranteed that the scenario I described cannot happen?

If Rebalance on Start remains disabled because it is still a Technology Preview feature, the HA resources from the failed node will have to be started on other nodes. Since Rebalance on Start won't be used to determine the placement, what criteria will be used to decide which node they start on?

This is the part that's a bit confusing to me.
 

Attachments

  • 1785318529428.png
    1785318529428.png
    122.3 KB · Views: 2
Last edited:
oktay454 said:
Right now, we're concerned that when we enable HA in our existing cluster, if a node fails, the VMs may end up being concentrated on a single node or drive one of the remaining nodes to 100% resource utilization. If that happens, the overloaded node could also become unresponsive, potentially causing a failover loop where the node taking over the HA resources also fails. As a result, the entire cluster could go down.
Click to expand...
If a system needs high availability, there must be enough redundant hardware to compensate such node failures across the cluster. How many cluster node failures should be compensated is a question of cluster size (i.e. what is possible with respect to storage, corosync, redundant hardware, etc.). Therefore, one should only make their HA resources require as much CPU time and memory as the cluster nodes minus the maximum amount of tolerated node failures can handle.

oktay454 said:
With the configuration shown in this screenshot, is it guaranteed that the scenario I described cannot happen?

If Rebalance on Start remains disabled because it is still a Technology Preview feature, the HA resources from the failed node will have to be started on other nodes. Since Rebalance on Start won't be used to determine the placement, what criteria will be used to decide which node they start on?
Click to expand...
The technology preview status does not correlate whether the described scenario can never happen, that is up to how much load is put on the cluster with respect to how many node failures happened vs. how many node failures can be tolerated.

The rebalance-on-start feature is only active if an HA resource is actually started. That is, if the HA resource was previously stopped and is requested to be started, the HA stack / CRS will first check whether there is a more suitable node than the current one and if there is one it will move there.

In case of a node failure and HA resource recovery, the CRS will recover the HA resources to available nodes according to the HA affinity rules and in the end pick the node that is deemed the most fitting node according the scheduler mode (basic / static load / dynamic load). In current practice, if there is enough resources left on the remaining cluster nodes, the CRS will pick suitable recovery nodes. But that is not possible if the cluster is overcommited in case of node failures.
 
Thank you very much.
Finally, I'm referring to the following scenario.

Let's assume the cluster has several nodes running at varying utilization levels. If one node fails, its HA virtual machines are recovered onto the remaining nodes — not concentrated onto a single node, but distributed according to the selected CRS scheduler mode.

For context on our setup: we have a single HA resource affinity rule containing all 8 nodes and all VMs in the cluster, with no priority values assigned between nodes.

My concern is more specific: when a recovery node is picked for a given HA resource after a node failure, does the CRS scheduler correctly account for the available capacity of each remaining node before placing that resource? Or could it, in some cases, place a resource onto a node that already has relatively high utilization instead of a node with more headroom?

For example, imagine an 8-node cluster where each node has 2 TiB of RAM. One node hosts a single large VM using 1 TiB of RAM. If that node fails, and the two candidate recovery nodes have 900 GiB and 1 TiB of RAM in use respectively, placing the 1 TiB VM onto the node that already has 1 TiB in use would bring that node to exactly 2 TiB — its full capacity, with zero headroom left for any other workload or spike — while the node with 900 GiB in use would still have roughly 1.1 TiB of headroom to absorb it comfortably.

This also raises a more basic question I'm not fully clear on: during HA recovery, is CRS itself the mechanism that selects the recovery node, or does HA recovery operate as a separate process that simply defers to CRS scoring/placement logic? The previous answer in this thread ("the CRS will recover the HA resources... and pick the node that is deemed the most fitting node") suggests CRS is directly involved, but I'd like to confirm whether CRS is the actual decision-maker here, or whether there's a distinct HA recovery mechanism that only consults CRS as an input.

My understanding is that "rebalance-on-start" and "automatic rebalance" only move already-running resources after the fact, based on load — they don't determine initial placement during HA recovery. So my real question is: does the CRS dynamic-load scheduler reliably pick the node with the most available capacity during recovery itself, or is there still a possibility that it picks a less optimal node and creates a tight/zero-headroom situation before automatic rebalance can correct it afterward?
 
You must log in or register to reply here.
Top Bottom
Back