[High Availability] Manually Invoke Cluster Resource Scheduling Migrations?

[ahayes]

Hello. I am relatively new to Proxmox VE but have been up and running with a 5 identical node cluster with Ceph storage and a remote PBS in production for about a year now and went through the excellent Croit online training a few months ago.

Up until now, I have been shuffling VMs around manually but I just finally got around to setting up High Availability on the PVE cluster. I have a simple setup with one group containing all nodes and all VMs set to that one group. It seems to work fine. I just went through an upgrade of all nodes one at a time with HA handling the migration of VMs when I rebooted each node (ha: shutdown_policy=migrate). The CRS static mode (crs: ha=static) also seems to work well in that scenario, distributing VMs throughout the cluster without issue. All good so far.

But because I have been doing these migrations manually up until now, and because I am blessed with an abundance of capacity in each node, the VMs are not currently well distributed. Instead I have ended up with all of them on a couple of the nodes. Since the HA system prefers to put things back where they were, I have not been able to take advantage of the CRS static mode to help me.

What I would like to be able to do is to manually invoke a redistribution of VMs in the cluster based on what the CRS would do. This would be useful to minimize the impact of any one node going down in the future, and would also help me test the CRS static mode in my environement with my VMs. Is this possible somehow?

 

[Alwin Antreich]

Hello. I am relatively new to Proxmox VE but have been up and running with a 5 identical node cluster with Ceph storage and a remote PBS in production for about a year now and went through the excellent Croit online training a few months ago.

Thank you.

What I would like to be able to do is to manually invoke a redistribution of VMs in the cluster based on what the CRS would do. This would be useful to minimize the impact of any one node going down in the future, and would also help me test the CRS static mode in my environement with my VMs. Is this possible somehow?

CRS [0] only kicks in on "service recovery", "ha group changes" and "stopped to started transition". IINM then setting the shutdown_policy to well, "shutdown", would trigger the reevaluation. Likely not what you'd want to do.

But what if you use the "basic" scheduler instead? The basic scheduler only counts how many resources under HA are on a particular node and selects the one with the lowest count. That could distribute the ha resources more evenly. And with enough resources on every node a congestion of a single node seems unlikely.

[0] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_crs

EDIT: basic scheduler - under the assumption that all VM & CT are under HA.

 

[fiona]

Hi,

But what if you use the "basic" scheduler instead? The basic scheduler only counts how many resources under HA are on a particular node and selects the one with the lowest count. That could distribute the ha resources more evenly.

For evenly configured guests/nodes, static behaves the same as basic. But if configurations differ between nodes or guests, static is better, because it looks at the configured RAM (and to a lesser extent CPU) and can make sure the configured resources are distributed more evenly between nodes. For example, if you have two heavy VMs and lots of smaller guests, basic has no idea that it should put the two heavy ones on different nodes.

 

[Alwin Antreich]

For evenly configured guests/nodes, static behaves the same as basic. But if configurations differ between nodes or guests, static is better, because it looks at the configured RAM (and to a lesser extent CPU) and can make sure the configured resources are distributed more evenly between nodes.

Exactly, that's why I suggested to switch to basic, as I assume not all resources are configured equally.

EDIT: I meant to say, I would assume it triggers a recalculation when switching.

 

[ahayes]

Hi again @Alwin Antreich! And thanks for jumping in @fiona.

I'm pretty sure I understand the differences between the two schedulers. Basic is counting "HA resources" (VMs and containers) and trying to have the same number of them on each node, regardless of how many system resources (CPU and RAM) they are using. Static is trying to balance them according to system resources being consumed so the nodes are more evenly used.

My question is more about if there is a way to force a CRS managed redistribution of VMs manually. I can shutdown a node to force a redistribution of VMs, but when the node comes back up they all get migrated back. I would like to just be able to say "Rebalance the cluster using the static scheduler to prioritize even system resource use and leave it like that until they need to be moved again, please." It would be like a run-once version of what I'm sure is being worked on for more dynamic rebalancing the way VSphere DRS can shuffle VMs around according to administrator configured priorities.

I only have a little over 60 VMs right now so I just shuffled them around manually today. But as I add new ones and reconfigure existing ones it would be nice to be able to auto-distribute them among the nodes for optimal system resource usage. CPU, RAM, and Network could all potentially be factored into the scheduler for consideration.

Anyway, it sounds like the answer is no for the moment. And that's fine. Thanks!

 

[fiona]

EDIT: I meant to say, I would assume it triggers a recalculation when switching.

It does re-calculate the usages, but the calculation is currently only used during start of a service (if ha-rebalance-on-start is set) and for services that need to be recovered or migrated.

Hi again @Alwin Antreich! And thanks for jumping in @fiona.

I'm pretty sure I understand the differences between the two schedulers. Basic is counting "HA resources" (VMs and containers) and trying to have the same number of them on each node, regardless of how many system resources (CPU and RAM) they are using. Static is trying to balance them according to system resources being consumed so the nodes are more evenly used.

My question is more about if there is a way to force a CRS managed redistribution of VMs manually. I can shutdown a node to force a redistribution of VMs, but when the node comes back up they all get migrated back. I would like to just be able to say "Rebalance the cluster using the static scheduler to prioritize even system resource use and leave it like that until they need to be moved again, please." It would be like a run-once version of what I'm sure is being worked on for more dynamic rebalancing the way VSphere DRS can shuffle VMs around according to administrator configured priorities.

I only have a little over 60 VMs right now so I just shuffled them around manually today. But as I add new ones and reconfigure existing ones it would be nice to be able to auto-distribute them among the nodes for optimal system resource usage. CPU, RAM, and Network could all potentially be factored into the scheduler for consideration.

Anyway, it sounds like the answer is no for the moment. And that's fine. Thanks!

You are correct. Unfortunately, there is no such feature to rebalance already started HA services right now.

 

[Evertos]

As many people looking to move away from a sertain product..
It would be nice if this is avialabel, is there a feature request ongoing regarding this?

DRS (Distributed Resource Scheduler) is a utility that balances computing workloads with available resources in a virtualized environment.

 

[fiona]

Hi,

As many people looking to move away from a sertain product..
It would be nice if this is avialabel, is there a feature request ongoing regarding this?

DRS (Distributed Resource Scheduler) is a utility that balances computing workloads with available resources in a virtualized environment.

yes, it is on the roadmap: https://pve.proxmox.com/wiki/Roadmap#Roadmap