Clarification related to HA Maintenance Mode/Affinity Rules

[Libero_AT]

We currently have a three node cluster, each node running Proxmox 9.x, with shared storage (over iSCSI). We are currently evaluating the behavior of HA-managed resources with Node/Resource affinity rules when Maintenance Mode is enabled.

We have failback enabled on every VM that is registered in the HA Manager. This would ensure that whenever a higher priority node becomes active, the VM will migrate (back) to it.

Our intention here is to be able to understand the underlying logic of the HA Manager in the following scenarios.

Scenario 1:
We have two VMs running on Node#3. We have added the two VMs to a "Keep together" resource affinity rule.
I now place Node #3 in maintenance mode. As expected, the VMs migrate together to the node with the least HA-managed running VMs (the crs mode is basic); let's say this was Node#2.
However, when maintenance mode was disabled on Node#3, the VMs did not migrate back, but instead stayed on Node#2.
Isn't this against the premise of the maintenance mode, which state that the VMs would return back to the node they were originally running on?

Scenario 2:
Again, consider the same VM's running on Node#3
The two VMs are added to a Node affinity rule (Node #2 and Node #3, equal priority, and Node#1 not included), but it is not strict.
The two VMs are also added to a "Keep separate" resource affinity rule. At this stage, both the VMs initially migrate, one each to Nodes #1 and #2, but since Node #3 has higher priority, one of the VMs migrates back from Node #1 to #3.
I now place Node #3 in maintenance mode. One would expect the VM running on Node#3 to migrate to Node#1, as per the premise of the maintenance mode.
Instead, the VM continues running on Node#3 (which is under maintenance mode). There are no existing HA-managed VMs on Node#1.

Could you explain the behavior in the above two cases?

 

[dakralex]

Welcome to the Proxmox forum, Libero_AT!

For scenario 1, it seems like that the current HA stack gives more priority that the resource affinity rule holds than whether it should migrate back to its maintenance node.

For scenario 2, I assume that the negative resource affinity rule (keep separate) was added a little before the node affinity rule, which made the VMs migrate to node1 and node2 before and only when the HA Manager read the node affinity rule migrated one of these VMs to node3 to comply with the node affinity rule. At least when I add both rules at the same time, there's only one migration for one VM to node2.

Nonetheless, for scenario 2 it seems like the HA Manager again prioritizes the affinity rules more strongly than whether the HA resource is on a maintenance node.

I will look into both!

Would be great to have a Bugzilla entry for this to track it independently, which can be created here [0].

[0] https://bugzilla.proxmox.com/enter_bug.cgi?product=pve&component=HA

 

[Libero_AT]

Hi Daniel, thank you for replying!

For scenario 2, I assume that the negative resource affinity rule (keep separate) was added a little before the node affinity rule, which made the VMs migrate to node1 and node2 before and only when the HA Manager read the node affinity rule migrated one of these VMs to node3 to comply with the node affinity rule.

You are right; we added the node affinity rule after the resource affinity rule in Scenario 2. My bad.

We would like to be able to concretely predict the migration of HA resources when a migration trigger is applied.
Hence, we are looking into if there's a simplified way of looking at the dynamics between the different aspects of the HA stack.

For instance, is it possible to definitively rank the following in an order of precedence?

• Resource affinity rules/Strict node affinity rules
• Non-strict node affinity rules
• Maintenance mode/Failback
• Cluster resource scheduler (rebalancing resources across nodes based on number of HA-managed active resources)

Thank you for your time.

 

[dakralex]

We would like to be able to concretely predict the migration of HA resources when a migration trigger is applied.
Hence, we are looking into if there's a simplified way of looking at the dynamics between the different aspects of the HA stack.

For instance, is it possible to definitively rank the following in an order of precedence?

In general, there shouldn't really be a precedence as all of those conditions should hold at the same time.

The rule verification system does dismiss many types of affinity rules, which cannot be determined to be resolvable at runtime, see [0].
There are still valid cases, which still could cause undesirable behavior to users, such as restricting an HA resource to only one node (cannot be recovered in case of failure or moved away from maintenance nodes) or negative resource affinity rules with as many HA resources as cluster nodes (a single failed or maintenance node would make the rule impossible to enforce). In these cases, admins should take great care that they think about the consequences these HA rules might have when nodes fail.

Nonetheless, the scenarios you described are bugs to me and I'm preparing a patch series to fix them and post here as soon as that's done.

Hope that clears things up, if you have any questions left, feel free to ask!

[0] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_rule_conflicts

 

[Libero_AT]

Hi Daniel, thanks for replying!

So, to conclude, the basic premise of maintenance mode, i.e.,

• If maintenance mode is activated, any HA-registered VM must migrate out of it
• If maintenance mode is disabled, the VMs that had migrated out of it, should migrate back to it*

will always be respected, irrespective of any affinity rules or other factors, as long as there are no conflicts?
*This is contingent to failback being set to true, right? In the case of failback set to false, we observe that the VMs do not migrate back.

In short, would you say the maintenance mode behavior is a hard constraint (like a strict affinity rule), as opposed to a soft constraint (like a non-strict node affinity rule, or the CRS)?

 

[dakralex]

So, to conclude, the basic premise of maintenance mode, i.e.,

• If maintenance mode is activated, any HA-registered VM must migrate out of it
• If maintenance mode is disabled, the VMs that had migrated out of it, should migrate back to it*

will always be respected, irrespective of any affinity rules or other factors, as long as there are no conflicts?

Not directly, this will only be respected if there are no affinity rules, which prohibit this behavior. For example, if the node affinity rule in your second scenario would have been strict, then the behavior would be correct as the HA resource on node3 does not have any other place to go but node3. In this scenario, this would mean that node3 never truly can disable its watchdog since it is unable to move all HA resources to other nodes.

I proposed a patch series [0] to remedy these situations (including your two scenarios) and that the HA Manager explicitly warns users in the log if the HA Manager cannot find a replacement node for an HA resource on a maintenance node. There's another Bugzilla entry [1] now, which aims to make this state, where a maintenance node still has HA resources on it, more visible to the user.

In short, would you say the maintenance mode behavior is a hard constraint (like a strict affinity rule), as opposed to a soft constraint (like a non-strict node affinity rule, or the CRS)?

Maintenance mode indicates that HA resources need to move to other nodes, because the node is marked as unavailable (similar to an offline node, though HA resources can still run fine on the maintenance node). Whether it really moves to another node is dependent whether enough nodes are available with all constraints (online nodes, node affinity rules, resource affinity rules) applied. So the maintenance mode is more like a trigger than a specific constraint, but in the usual circumstances, maintenance mode always means that HA resources need to be moved away and come back as soon as the node is not in maintenance anymore.

[0] https://lore.proxmox.com/pve-devel/20260422100035.232716-1-d.kral@proxmox.com/
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7516

 

[Libero_AT]

Hi Daniel, thanks for the explanation, and the proposed fixes.
I noticed your suggestion to prioritize returning to the original node after maintenance mode, even when failback:false. That's interesting!

I have an additional question; this is not related to maintenance mode.
When the HA manager needs to choose between two nodes with identical scores after evaluating all factors (like affinity rules and the CRS scheduler), is the destination chosen based on the alphabetical order of the node names?

The only mention of prioritising alphabetical order in the HA-Manager documentation is this specific case. I wanted to make sure if this is a general principle.

If the HA resources of a positive resource affinity rule are currently running on different nodes, the CRS will move the HA resources to the node, where most of them are running already. If there is a tie in the HA resource count, the node whose name appears first in alphabetical order is selected.

 

[dakralex]

I noticed your suggestion to prioritize returning to the original node after maintenance mode, even when failback:false. That's interesting!

I proposed to change that since the failback flag's description [0] explicitly states it's only concerned about moving back to a higher priority node class, but for now it's a proposal .

When the HA manager needs to choose between two nodes with identical scores after evaluating all factors (like affinity rules and the CRS scheduler), is the destination chosen based on the alphabetical order of the node names?

Exactly, in the end if two nodes have the exact same score (e.g. exactly the same amount of guests on each node with the 'basic' scheduling mode), then the node alphabetically first is chosen to be the destination.

Currently, the same is true for the order how HA resources are processed, so if vm:101 and vm:102 can both be placed on the same nodes (e.g. node3 and node4), then vm:101 will be put on node3 and vm:102 on node4. Though this might change in the future as node placement strategies are improved upon further.

Feel free to create a Bugzilla entry [1] if the HA documentation should elaborate on such things, so that more users benefit from the additional information in the documentation!

[0] https://pve.proxmox.com/pve-docs/api-viewer/index.html#/cluster/ha/resources/{sid}
[1] https://bugzilla.proxmox.com/enter_bug.cgi?product=pve&component=HA