Questions about the dynamic CRS

[jsterr]

@dakralex are there plans or ideas about some additions for usecases like:

* Company has a multi-room setup with for example 2 nodes in room1 and 2 nodes in room2 and a quorum node in room 3
* Theres a ha node affinity rule for room1 (node restrict to node1 and node2) for vms that should run in room 1
* Theres a ha node affinity rule for room2 (node restrict to node3 and node4) for vms that should run in room 2
* Ressource affinity rule for splitting application-clusters like (active active db-cluster) to run on different hosts (keep separate)

This is already possible. But what customers have asked a few times is can it be set so vms should stay in their specific rooms BUT if a room fails, they should automatically be allowed to run on the other room as well -> like automatically disabling the node restriction.

Like situation aware cluster ressource schedules? or maybe some kind of feature to modify CRS-rules on specific events? so the rules stay the same but can be automatically adjusted on specific szenarios?

Currently im telling the customers they can always manually change rules in that case, but it lacks the automatism.

Whats your opionion on that?

 

[dakralex]

Hi, sorry for the late reply!

This is already possible. But what customers have asked a few times is can it be set so vms should stay in their specific rooms BUT if a room fails, they should automatically be allowed to run on the other room as well -> like automatically disabling the node restriction.

At least for two rooms this is already possible with non-strict node affinity rules as these will fallback to the other available cluster nodes if none of the specified nodes are available anymore. However, an HA resource cannot be specified in a (positive/negative) resource affinity rule and a node affinity rule with multiple node priority classes (e.g. for more than two rooms) at the same time at the moment.

The main reason why this is not allowed at the moment is because the HA stack does check whether these affinity rules can cause any inconsistencies and whether these affinity rules can be applied deterministically at all times. This would be rather computationally expensive to do with multiple node priority classes as this would need cross-checking with all permutations of nodes being online or offline.

However it would be interesting to investigate in being able to have either an event-based system, which allows to change the behavior of the affinity rules in a controlled way, or decompose the current node affinity rules in a better way to be able to compute the feasibility of multi-priority class node affinity rules in conjunction with resource affinity rules more efficiently to be able to accommodate such scenarios.

Do you have more insights to more concrete examples about these use cases?

 

[jsterr]

Do you have more insights to more concrete examples about these use cases?

Hi Daniel! Some cases that do not work or might be cool adding them:

Example 1: A resource affinity rule with “Keep Separate” for VM1 and VM2, combined with an HA node affinity rule (without “Strict”) on node1 and node2 (checked), but not checked for node3. VM2 cannot start if node2 fails, but it could potentially run on node3 as well. However, it does not start there → the VM enters the “recovery” state and remains there (the VM stays on the shut-down node).

Example 2: HA VM node affinity set to "Strict" on node3 only → Reboot the node → The node gets stuck in maintenance mode in the LRM and nothing happens. Possibly check for strict rules (limited only to one node) when trying to activate maintenance mode and provide feedback if maintenance mode cannot be executed, or fall back to the shutdown policy (although migrating the shutdown policy could also lead to problems). If this happens maybe offering a shutdown as a possibility.

In generell I get a lot of feedback of wanting the maintenance mode inside the web-ui. But I see, there are still some edge cases that might need some fix to be fully functional (like example 2)

This and having a possibility to automatically set / add all vms to ha or having a default that can be changed to "ha always on = yes". Although I know especially when people want/should/must use ha-node-affinity or/and ha-ressource-affinity rules for example multi-room setups, a "ha always on" still does not fix that, as ha ressources would still need to be assigned manually to the rules. but it would be still a benefit, because forgetting a vm to get put into HA wouldnt be a thing anymore. Of course there are other edge-cases like autoadding a vm to ha, which might still have a local ressource in it (and might also keep it and should get restricted) - but thats why it should not be default = yes.

It would be cool to have the flexibility in adding/removing and managing ha ressources like you have when doing "bulk-actions" there you have a list of vms that are shown, you can filter, you can search etc. having this in ha would be a huge win as you could then just use the "checkbox" that marks all items and add them to ha in one go.Something like this (ai draft)

 

[dakralex]

Hi Jonas!

Example 1: A resource affinity rule with “Keep Separate” for VM1 and VM2, combined with an HA node affinity rule (without “Strict”) on node1 and node2 (checked), but not checked for node3. VM2 cannot start if node2 fails, but it could potentially run on node3 as well. However, it does not start there → the VM enters the “recovery” state and remains there (the VM stays on the shut-down node).

Good catch! There is a patch for this already on the mailing list [0], I will send a new revision soon so this gets resolved!

Example 2: HA VM node affinity set to "Strict" on node3 only → Reboot the node → The node gets stuck in maintenance mode in the LRM and nothing happens. Possibly check for strict rules (limited only to one node) when trying to activate maintenance mode and provide feedback if maintenance mode cannot be executed, or fall back to the shutdown policy (although migrating the shutdown policy could also lead to problems). If this happens maybe offering a shutdown as a possibility.

This does only happen with the "migrate" shutdown policy and is documented in the documentation for the HA Manager [1].

It might be plausible to make the LRMs revert to something like the 'shutdown' or 'failover' shutdown policy, where these HA resources are just stopped on the node and cannot be recovered to another node after some timeout, but this could also be unexpected behavior for users. Feel free to create a Bugzilla entry for that if none exists already.

User-visible information about the progress of the going-into-maintenance-mode state seems like a good idea for this as well, I pointed it out in this Bugzilla entry [2], feel free to add any ideas to it!

[0] https://lore.proxmox.com/pve-devel/20260422100035.232716-8-d.kral@proxmox.com/
[1] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#_migrate
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=7516

 

[dakralex]

This and having a possibility to automatically set / add all vms to ha or having a default that can be changed to "ha always on = yes". Although I know especially when people want/should/must use ha-node-affinity or/and ha-ressource-affinity rules for example multi-room setups, a "ha always on" still does not fix that, as ha ressources would still need to be assigned manually to the rules. but it would be still a benefit, because forgetting a vm to get put into HA wouldnt be a thing anymore. Of course there are other edge-cases like autoadding a vm to ha, which might still have a local ressource in it (and might also keep it and should get restricted) - but thats why it should not be default = yes.

It would be cool to have the flexibility in adding/removing and managing ha ressources like you have when doing "bulk-actions" there you have a list of vms that are shown, you can filter, you can search etc. having this in ha would be a huge win as you could then just use the "checkbox" that marks all items and add them to ha in one go.

Generally, these sound like good ideas! Feel free to create separate Bugzilla entries for the separate issues:

• automatically add new guests as HA resources by enabling some cluster-wide check (Though new guests can be automatically added one-by-one by enabling the "Add to HA" in the web interface and through the API / CLI endpoints)
• A bulk HA resource management window/modal/... where it's easier to manage the addition/removal of HA resources and their configs in bulk more easily, though the addition/removal part and config editing part are strictly speaking two separate concerns as well.
• Adding new HA resources to certain affinity rules.