Maximum number of VMs per Proxmox server?

voidindigo

New Member
Sep 18, 2018
3
0
1
52
Sorry if this is a repeat or in the FAQ somewhere, but I'm very new to Proxmox and I couldn't find it.

Is there a max number of VMs I can configure on a Proxmox server? Our test environment will need 1000 VMs, probably ~200 VMs per server, with a cluster of 5 or 6 Proxmox servers.

Is that a viable configuration (assuming we have the hardware to handle it, of course)?

Thank you, and thank you for some great software!
-
Scott
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
3,953
905
163
South Tyrol/Italy
shop.maurer-it.com
Is there a max number of VMs I can configure on a Proxmox server? Our test environment will need 1000 VMs, probably ~200 VMs per server, with a cluster of 5 or 6 Proxmox servers.

Is that a viable configuration (assuming we have the hardware to handle it, of course)?

Yes, should be no problem. The only restriction regarding VM counts are two:
* The real time replicated filesystem has a maximal size of 32 MB, currently, and a normal config (no snapshots) has about 200 to 300 bytes, so that leaves a theoretical limit of ~100.000 VM configs, (there are a few other files so your mileage may vary, but that is roughly the order of magnitude you could get). If you use a lot of VM/CT snapshots the config files get bigger (roughly multiply base size with number of snapshots).
* The HA manager is currently able to manage about 3000 services concurrently (no hard limit, but we have the status of all services in one file, which max size is capped at 512KB)

So 1000 VMs should really not be a problem. In fact, I know production setups handling school websites which has >1500 CT on two PVE nodes (plus one quorum node).
 

voidindigo

New Member
Sep 18, 2018
3
0
1
52
Right, and memory allocated to a VM is always "in use" to some degree. I'm familiar with VMs in general, but each system is a little different. Also, I'm really interested in the Containers in Proxmox, as an alternative way to deploy software in a low-overhead way. Some of our systems require a dedicated environment and will need the full VM, but I think some could be stripped down to software-only and I may be able to deploy them into a Container, which should give me a lot more instances on a single VM server.

Thanks again
 

Greatsamps

Member
Sep 25, 2017
29
0
6
40
* The HA manager is currently able to manage about 3000 services concurrently (no hard limit, but we have the status of all services in one file, which max size is capped at 512KB)

Just wondering if there has been any change in this in recent releases?
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
3,953
905
163
South Tyrol/Italy
shop.maurer-it.com
Just wondering if there has been any change in this in recent releases?
There were some internal discussion about increasing this for the next major release (the changes are not exactly backward compatible).

Is 3000 service not enough for you? Can you please share a bit of information about your use case?
 

Greatsamps

Member
Sep 25, 2017
29
0
6
40
Well, 3000 is quite a lot!

Our use case is as follows. We run a SaaS application that interfaces into a 3rd Party application which is very badly behaved! As such we need to keep it very tightly constrained resource-wise. We actually have the application packaged up into a docker container for ease of orchestration which we are deploying onto a very lightweight VM that we have created as a docker host.

So I imagine at this stage you are saying, why not use LCX containers? Well, we have tried running large quantities of these applications on a single docker instance. Not sure exactly what is going on inside the 3rd Party application but it generates a lot of threads, some of which have elevated priorities and once you get past a certain number, the entire machine becomes thread/resource-starved, even if you set CPU limits on the container. Our conclusion is that the kernel can't manage all these competing threads as well as having them isolated at the hardware level and managed by a hypervisor.

So our use case is really container scale VM's within the cluster. Right now we have 2,500 of these containers which we are looking to migrate onto the cluster. My initial query was knowing if we will need to build out a second cluster once we hit the 3k mark.
 

guletz

Famous Member
Apr 19, 2017
1,458
226
83
Brasov, Romania
We actually have the application packaged up into a docker container for ease of orchestration which we are deploying onto a very lightweight VM that we have created as a docker host.

Hi,

With your not so much details about your case, maybe you can try to to split your load to many "lightweight VM" using a Load-Balancing in front of your clients(so you will not "get past a certain number"/each "lightweight VM")

Good luck / Bafta!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!