Maximum number of VMs per Proxmox server?

[voidindigo]

Sorry if this is a repeat or in the FAQ somewhere, but I'm very new to Proxmox and I couldn't find it.

Is there a max number of VMs I can configure on a Proxmox server? Our test environment will need 1000 VMs, probably ~200 VMs per server, with a cluster of 5 or 6 Proxmox servers.

Is that a viable configuration (assuming we have the hardware to handle it, of course)?

Thank you, and thank you for some great software!
-
Scott

 

[t.lamprecht]

Is there a max number of VMs I can configure on a Proxmox server? Our test environment will need 1000 VMs, probably ~200 VMs per server, with a cluster of 5 or 6 Proxmox servers.

Is that a viable configuration (assuming we have the hardware to handle it, of course)?

Yes, should be no problem. The only restriction regarding VM counts are two:
* The real time replicated filesystem has a maximal size of 32 MB, currently, and a normal config (no snapshots) has about 200 to 300 bytes, so that leaves a theoretical limit of ~100.000 VM configs, (there are a few other files so your mileage may vary, but that is roughly the order of magnitude you could get). If you use a lot of VM/CT snapshots the config files get bigger (roughly multiply base size with number of snapshots).
* The HA manager is currently able to manage about 3000 services concurrently (no hard limit, but we have the status of all services in one file, which max size is capped at 512KB)

So 1000 VMs should really not be a problem. In fact, I know production setups handling school websites which has >1500 CT on two PVE nodes (plus one quorum node).

 

[voidindigo]

That's perfect! Thank you for the quick reply!
-
Scott

 

[spirit]

Note that each vm , even idle, use a litle bit host cpu for each vcpu. I m running 200vms on 2x12 cores intel without problem

 

[voidindigo]

Right, and memory allocated to a VM is always "in use" to some degree. I'm familiar with VMs in general, but each system is a little different. Also, I'm really interested in the Containers in Proxmox, as an alternative way to deploy software in a low-overhead way. Some of our systems require a dedicated environment and will need the full VM, but I think some could be stripped down to software-only and I may be able to deploy them into a Container, which should give me a lot more instances on a single VM server.

Thanks again

 

[Greatsamps]

* The HA manager is currently able to manage about 3000 services concurrently (no hard limit, but we have the status of all services in one file, which max size is capped at 512KB)

Just wondering if there has been any change in this in recent releases?

 

[t.lamprecht]

Just wondering if there has been any change in this in recent releases?

There were some internal discussion about increasing this for the next major release (the changes are not exactly backward compatible).

Is 3000 service not enough for you? Can you please share a bit of information about your use case?

 

[Greatsamps]

Well, 3000 is quite a lot!

Our use case is as follows. We run a SaaS application that interfaces into a 3rd Party application which is very badly behaved! As such we need to keep it very tightly constrained resource-wise. We actually have the application packaged up into a docker container for ease of orchestration which we are deploying onto a very lightweight VM that we have created as a docker host.

So I imagine at this stage you are saying, why not use LCX containers? Well, we have tried running large quantities of these applications on a single docker instance. Not sure exactly what is going on inside the 3rd Party application but it generates a lot of threads, some of which have elevated priorities and once you get past a certain number, the entire machine becomes thread/resource-starved, even if you set CPU limits on the container. Our conclusion is that the kernel can't manage all these competing threads as well as having them isolated at the hardware level and managed by a hypervisor.

So our use case is really container scale VM's within the cluster. Right now we have 2,500 of these containers which we are looking to migrate onto the cluster. My initial query was knowing if we will need to build out a second cluster once we hit the 3k mark.

 

[guletz]

We actually have the application packaged up into a docker container for ease of orchestration which we are deploying onto a very lightweight VM that we have created as a docker host.

Hi,

With your not so much details about your case, maybe you can try to to split your load to many "lightweight VM" using a Load-Balancing in front of your clients(so you will not "get past a certain number"/each "lightweight VM")

Good luck / Bafta!