Load balancing and future features

[mgiammarco]

Hello,
Proxmox VE is a great product and with last version is even bigger.
I have some big customers interested but only two important features are missing:
- load balancing
- maintenance mode

I have seen the second one is on the roadmap, but can someone tell me if load balancing of vm will be implemented?

I think it is a feature missing so more important than, for example, btrfs support (you already support ceph, zfs, nfs and so on)

Thanks for any info.
Mario

 

[spirit]

Hi,

>>have seen the second one is on the roadmap, but can someone tell me if load balancing of vm will be implemented?

the main problem is that currently to do balancing, proxmox need to be router/gateway (at least for inbound traffic).
not too much complex with a single host, but with a cluster, we need to manage failover of the ip, or implement anycast routing.
I'm currently working on vxlan+evpn bgp to have a true sdn and anycast routing (something like vmware nsx), After that it'll be easier to implement new network features. (dhcp, balancing,....)

 

[mgiammarco]

Sorry reading your reply I understand I have explained myself terribly wrong.
I was meaning load balancing of the servers load, migrating automatically virtual machines. And when you start maintenance mode, moving all vm of the server to other servers respecting load.

 

[spirit]

oh ok, sorry. vm scheduling. We are just talking about that today on dev mailing list

So, I don't think for proxmox 6.0, but maybe in proxmox 6.X it could be possible.

 

[alexskysilk]

I was meaning load balancing of the servers load, migrating automatically virtual machines. And when you start maintenance mode, moving all vm of the server to other servers respecting load.

You're looking for DRS in vmware parlance. This has been a major ask for as long as I can remember but it doesnt seem to be either high priority for the proxmox team or its just too difficult (or both.) this is one of definite use cases for vsphere over proxmox.

It is possible to write such functionality using your language of choice. I recall reading some user(s) that have done just so based on server load, cpu utilization, phase of the moon, etc- but its easier and more effective to just have more spare capacity.

 

[mgiammarco]

You're looking for DRS in vmware parlance. This has been a major ask for as long as I can remember but it doesnt seem to be either high priority for the proxmox team or its just too difficult (or both.)
...but its easier and more effective to just have more spare capacity.

As a developer I suppose it was very difficult but now they have improved and many features needed to simulate DRS are now on place.
I know that with enough ram all problem are solved but in my specific case:
- when I add more ram users of the cluster add immediately more VMs and I have solved nothing;
- at least in maintenance use case when I power off a server with many VMs on it I would like that machines are migrated intelligently on remaining servers. It is severely limiting to not have a "maintenance mode"

Thanks,
Mario

 

[alexskysilk]

As a developer I suppose it was very difficult but now they have improved and many features needed to simulate DRS are now on place.

I'm curious; what feature(s) are you referring to? It may be possible to leverage them using external tools.

As to maintenance mode, you can already accomplish this functionality using HA groups. all you do is remove a node from the respective HA group and all containers assigned to said group will migrate to other available nodes.

 

[mgiammarco]

Starting from the end, thanks for precious hint.
Now replying to your first question about features: they have already an HA infrastructure that can do semiautomatic migration. You can assign priority to nodes. At this point you miss only an heuristics that, for example, dynamically modify priorities of nodes and with some tricks you can emulate DRS. It is just a quick idea to show that 80% of infrastructure to support DRS is already there.

 

[alexskysilk]

Bulk migration # semi automatic.

DRS is a deceptively simple sounding concept; practically it is difficult and one size does not fit all. Luckily for you, the api is simple to use and metrics to generate the heuristics you're referring to are readily available- if you can define the algorithm by which you want to act on this the actual programming would be relatively simple, and I'm pretty certain others have done so already. The trouble is that the time you'll spend fine tuning the rules would be greater then any benefit you'll likely to realize

 

[gurubert]

We created a script that is able to balance according to RAM usage. It is also able to evacuate a Proxmox node's VMs to the other nodes based on RAM usage.

# ./proxmox_migrate.py --help
usage: proxmox_migrate.py [-h] -u USERNAME -p PASSWORD [-n] [-d] [-w]
                          {evacuate,balanceram,migrate} ...

optional arguments:
  -h, --help            show this help message and exit
  -u USERNAME, --username USERNAME
  -p PASSWORD, --password PASSWORD
  -n, --dryrun
  -d, --debug
  -w, --wait

available commands:
  {evacuate,balanceram,migrate}
                        call "subcommand --help" for more information
    evacuate            evacuate first host, migrate VMs to other hosts
    balanceram          balance VMs over all hosts based on RAM usage
    migrate             migrate one VM to dest

You can find the code here: https://github.com/HeinleinSupport/proxmox-tools
It is a Python script that uses the proxmoxer module.

 

[phin]

This is awesome. Would also be great to do something like this based on CPU load.

 

[cvk98]

I have created a simple load balancer, try it out. He is far from perfect, but he copes with his task.
https://github.com/cvk98/Proxmox-memory-balancer

 

[cvk98]

Wrote a script for maintenance mode.
https://github.com/cvk98/Proxmox-host-maintenance-mode

 

[Elfy]

I have created a simple load balancer, try it out. He is far from perfect, but he copes with his task.
https://github.com/cvk98/Proxmox-memory-balancer

Looks great, using it now. Thanks for maintaining it.

 

[cvk98]

In the future, I will devote less time to it, as I am writing a new version. It will have to work constantly and maintain the specified RAM load balance. Follow the news at the link https://github.com/cvk98/Proxmox-load-balancer . The best thanks from users are the stars on github =)

 

[Elfy]

Followed and starred! Thank you for the hard work!

 

[cvk98]

I wrote a new script that allows you to keep the cluster in a balanced state all the time.
https://github.com/cvk98/Proxmox-load-balancer
I wanted to add a witty phrase, but when translated it loses its meaning =)

 

[spirit]

I'm currently working on a load balancer / vm scheduler (something like drs 2.0 from vmware with vm health/pressure metrics with multi-dimensionnal metrics support). It should be ready in coming months. (I need to add more stats to pvestatd first).

 

[cvk98]

I'm currently working on a load balancer / vm scheduler (something like drs 2.0 from vmware with vm health/pressure metrics with multi-dimensionnal metrics support). It should be ready in coming months. (I need to add more stats to pvestatd first).

Don't bother, I've already written it =)

 

[Uplink]

I'm currently working on a load balancer / vm scheduler (something like drs 2.0 from vmware with vm health/pressure metrics with multi-dimensionnal metrics support). It should be ready in coming months. (I need to add more stats to pvestatd first).

spirit are you developing a feature for integration into Proxmox itself?