[Tool] vman for managing PVE clusters

robhost

Active Member
Jun 15, 2014
224
9
38
Dresden
www.robhost.de
Hi,

we've built a helper tool for PVE clusters called vman which comes with useful functions which PVE is missing, for example:

* auto balance VMs in a PVE cluster depending on used and free memory
* flushing an entire PVE node (i.e. for updates/reboots)
* flushing only HA VMs (for updates)

vman will also provide IO stats for each VM in the next release, we'll upgrade the repo soon.
Feedback and contributions are welcome :)

Available on Github: https://github.com/robhost/pve_vman
 
Just a comment. Wouldn't it be smarter to directly communicate with the REST API instead of using pvesh? The obvious benefit is that vman does not need to run on pve nodes. There is also a python API for proxmox available on github which seems to be actively maintained: https://github.com/swayf/proxmoxer
 
Yes, using the REST-API would be a good approach so it could be used from anywhere.

However, there are some drawbacks with the API. The API is quite slow and for some information it is necessary to send multiple requests, like config details of all VMs. This scales very badly and quickly gets unusable. Of course this is true no matter how one uses the API either directly or via pvesh. Also, some information are not accessible via the API, like IOPS (We'll try to get patch ready for extending the QEMU stats, though).

Apparently, the best approach would be to enhance the API with the missing endpoints (like config for all VMs), so the API can be used directly for example via proxmoxer. Unfortunately, we are not familiar with perl development, so it might take some time for us to get proper patch ready.

As we had a fast growing cluster to manage we decided to take the shortcut of getting the information from the config files directly. Of course, this has the disadvantage that vman has to run on a node of the cluster so it can access this files. On the other hand this is much faster and scales much better.
 
On the other hand this is much faster and scales much better.
I do not think pvesh is much faster that directly use the API since pvesh uses the API as well. For the scaling assumption I tend to disagree as well since running on a pve node means you are restricted to one cluster while using the API on a separate server could scale to multible clusters.
 
Since pvesh uses the API as well the same applies to pvesh also.
to be exact, pvesh uses the same perl codepath as the api, without using the http path
 
Yes, as I said, pvesh is not faster than using the API directly. As I read my post again, I notice that I missed to make the point why we decided to use pvesh and not the API directly.

Since we access the files directly it is necessary to run the tool on a node of the cluster itself. Because of that, there is IMHO no advantage in using the API directly. pvesh has the advantage of not requiring any authentication data, though.
 
Could you please elaborate on how to fetch hundreds of VM configs a timely manner using the current HTTP-API? We haven't found a way to do that.

So for the time until we have developed a proper patch for extending the API for a clusterwide VM config endpoint, we read the config directly from /etc/pve.
 
Could you please elaborate on how to fetch hundreds of VM configs a timely manner using the current HTTP-API? We haven't found a way to do that.

So for the time until we have developed a proper patch for extending the API for a clusterwide VM config endpoint, we read the config directly from /etc/pve.

requesting 500 VM configs from a single node with 500 separate connections takes about 4s here (using curl)

if you use a single re-used connection with curl, this drops down to 0.55s

directly reading the 500 config files from pmxcfs takes 0.18s

requesting 500 configs each from three clustered nodes takes a little over 2s in total, with a single curl command with keep-alive. if you request all 1500 from a single node (which means that 1000 of those requests are internally proxied to the other two nodes), it takes around 4s.

I think all of this is still rather reasonable, and migrating that many VMs takes a lot longer anyhow so it is for sure not the bottle neck here ;) if you get vastly worse results, you are either doing something wrong/suboptimally (like opening a new connection for each request), or your network latency is too high if you are requesting things remotely..
 
requesting 500 VM configs from a single node with 500 separate connections takes about 4s here (using curl)

if you use a single re-used connection with curl, this drops down to 0.55s

directly reading the 500 config files from pmxcfs takes 0.18s
This is also my experience. What normally takes time is working with the data and do the appropriate operation afterwards.
 
requesting 500 VM configs from a single node with 500 separate connections takes about 4s here (using curl)

if you use a single re-used connection with curl, this drops down to 0.55s

directly reading the 500 config files from pmxcfs takes 0.18s

requesting 500 configs each from three clustered nodes takes a little over 2s in total, with a single curl command with keep-alive. if you request all 1500 from a single node (which means that 1000 of those requests are internally proxied to the other two nodes), it takes around 4s.

I think all of this is still rather reasonable, and migrating that many VMs takes a lot longer anyhow so it is for sure not the bottle neck here ;) if you get vastly worse results, you are either doing something wrong/suboptimally (like opening a new connection for each request), or your network latency is too high if you are requesting things remotely..

Thanks for your suggestions and the numbers. Reusing the connection is indeed much better.

So I stand corrected. Using the API directly (e.g. with proxmoxer) would be totally possible for the task at hand. :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!