How do you enforce per-pool (tenant) resource quotas for CPU / RAM / disk in Proxmox?

[Fearless-Grape5584]

I’m looking for a practical way to enforce tenant-level resource limits in Proxmox.

My current approach is to use Pools + RBAC to isolate tenant resources and delegate operations to a pool admin (create/delete VMs, start/stop, snapshots, backups, etc.) only within their own pool.
However, the obvious problem is that a pool admin can still consume as much CPU / RAM / disk as they want, which can impact other tenants.

What I’m trying to achieve is something like:

• Enforce hard limits per pool (CPU / RAM / disk), or
• Prevent VM creation (or require approval) once a pool exceeds its allocated capacity, or
• Hook into the event before a pool admin spins up VMs (e.g., a pre-create / pre-start guardrail) to enforce policy, or
• If Proxmox can’t do this natively, what’s the common “real-world” way people handle this?

How do you manage tenant-level capacity in practice? For example:

• separate clusters per tenant,
• external guardrails (API / IaC) that check quotas before creating VMs,
• monitoring + policy / chargeback,
• any other proven pattern.

Note: I’m not posting the full build steps here because it becomes quite long.
If anyone needs context on the exact setup, I documented the manual build process in my repo (I’m the author):
https://github.com/zelogx/proxmox-msl-setup-basic
The step-by-step instructions are in `build-instruction.md`.

Thanks in advance — this feels like a common multi-tenant pain point, so shared approaches would likely help others as well.

 

[bl1mp]

Hi,
depending on what you would like to archive, you can try to implement a solution via hookscripts. ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_hookscripts )

But a more suitable solution might be the Proxmox Datacenter Manager + multiple Cluster.

From the perspective of provisioning:
CPUs can be used shared,
KSM can save on RAM,
Storage quotas can usually be implemented directly on the storages.
Resource Groups are cluster wide, so usage stats needs to respect that.
Real Multi-tenancy also needs to consider anonymous neighbor tenants, which will be a nasty setup with multiple authentication realms.

BR, Lucas

 

[Fearless-Grape5584]

Hi,
depending on what you would like to archive, you can try to implement a solution via hookscripts. ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_hookscripts )

But a more suitable solution might be the Proxmox Datacenter Manager + multiple Cluster.

Hi Lucas,

thanks — that’s a very helpful perspective.

I agree that PDM + multiple clusters can be a solid approach when you want to separate tenants at the cluster boundary.

In my case, I’m exploring a different direction: squeezing multi-tenant-style delegation out of a single Proxmox server (or a single cluster) by slicing it with Pools + RBAC, and then adding some kind of preventive guardrail before provisioning.

So your pointer to hookscripts is especially useful for what I’m trying to achieve. I’ll dig into that path.

BR,
Masa

 

[LnxBil]

Prevent VM creation (or require approval) once a pool exceeds its allocated capacity, or

Storage capacity is easy if the underlying storage supports it. In the end you need a storage for each tenant, so that the configured storagr has the quota. There is (at the moment) no way to configure restrictions just based on some logic in PVE. The pools are not a resource group yet.

A few years back, I also played around with this idea. In a ZFS setup, you just create a new dataset on the CLI that has the quota limit, add it to PVE and assign it to the pool that the tenant should use. RBAC on the pool ensures that you can only create the VM on that storage. The same logic may apply to CEPH, as you can just create a new pool.

 

[Fearless-Grape5584]

Yeah, exactly – that’s the point.

Right now I do NOT have any real tenant-level quota for CPU or RAM, and that’s
exactly why I opened this thread.

In my current design I can isolate tenants with Pools + RBAC and separate networks
with SDN, and I can enforce hard limits only on storage (per-tenant LUN/dataset/pool
with quota). But CPU/RAM are still basically “best effort” – if a pool admin creates
too many big VMs, they can still hurt other tenants.

So I’m not presenting a solved setup here, I’m trying to learn how people handle this
gap in real-world Proxmox environments.

 

[Raph123FR]

Yeah, exactly – that’s the point.

Right now I do NOT have any real tenant-level quota for CPU or RAM, and that’s
exactly why I opened this thread.

In my current design I can isolate tenants with Pools + RBAC and separate networks
with SDN, and I can enforce hard limits only on storage (per-tenant LUN/dataset/pool
with quota). But CPU/RAM are still basically “best effort” – if a pool admin creates
too many big VMs, they can still hurt other tenants.

So I’m not presenting a solved setup here, I’m trying to learn how people handle this
gap in real-world Proxmox environments.

Do you find any solution for that ? I'm interested in performing the same procedure

 

[Fearless-Grape5584]

Admission control gets you part of the way. Here's what I found when I looked into this.
CPU and memory are workable. Hook pre-start and pre-migrate, read a per-pool quota, sum the allocations of the running VMs in that pool, and fail the start if the request would exceed the limit.
Disk is where it breaks down. You can cap the sum of provisioned virtual sizes, but that is a reservation limit, not a usage limit. With thin provisioning the actual consumption is unrelated to it. On LVM-thin I also could not find a reliable way to attribute snapshot space back to individual VMs, so any pool total you compute will drift from what the storage really uses.
Also worth noting: this only enforces at start and migrate, so runtime usage is still uncontrolled. I did not take it further than a single host.

Out of curiosity, what scale are you running? Number of tenants and whether it's a cluster would change the answer quite a bit.

 

[Raph123FR]

Admission control gets you part of the way. Here's what I found when I looked into this.
CPU and memory are workable. Hook pre-start and pre-migrate, read a per-pool quota, sum the allocations of the running VMs in that pool, and fail the start if the request would exceed the limit.
Disk is where it breaks down. You can cap the sum of provisioned virtual sizes, but that is a reservation limit, not a usage limit. With thin provisioning the actual consumption is unrelated to it. On LVM-thin I also could not find a reliable way to attribute snapshot space back to individual VMs, so any pool total you compute will drift from what the storage really uses.
Also worth noting: this only enforces at start and migrate, so runtime usage is still uncontrolled. I did not take it further than a single host.

Out of curiosity, what scale are you running? Number of tenants and whether it's a cluster would change the answer quite a bit.

Thank you for this additional information.

This is a new topic for us, and I’m currently evaluating the solution to see if it meets our needs as part of a VMware migration, so I haven’t done any further work on this yet.

Currently, we’re looking at approximately 5,000 VMs spread across different clusters, each dedicated to a specific technology (Windows, Linux, Oracle, others...). clusters can range from 5 to 15 nodes.

 

[Johannes S]

Never used or even tested it but maybe this covers your needs?
https://multiportal.io/ is a third-party software which aims to provide multi-tenant support to ProxmoxVE.

Despite that your scale might be that large that Openstack ( no personal experience either on my part) might be a better fit but it also has larger complexity as far I know.

 

[Raph123FR]

Thanks, I'll take a look ! We're still trying to avoid using too much "homemade or license" software however with Proxmox Manager or Pegaprox, I think we'll be able to handle cluster management.

OpenStack was within our scope, but the technical gap would be too big, and we don't have enough staff.

Right now, I'd like to properly segment my environments using RBAC + CPU, RAM, and STORAGE resources limitations since we grant certain clients access rights to the VMs.

I’ll continue my testing, and I’ll be sure to let you know if I find anything !

 

[Johannes S]

One thing I would consider is the availability of support. While the Proxmox Datacenter Manager (despite it's still limited feature set compared to vcenter) is covered by Proxmox support beginning with basic subscription level I'm not aware of any support offering for Pegaprox. On the other hand at least one of Pegaprox developers ( @gyptazy ) works at Proxmox Partner Credativ, maybe they are able to provide support? Asking them shouldn't hurt I guess
mutliportal has support included in their subscription, although their price might be a bit too much for your budget. They have custom offers for large scale deployments though according to their website: https://multiportal.io/pricing/

 

[alexskysilk]

A couple of comments:

1. Since you're deploying client assets in a cluster, there are two components to consider- first, what is your target overprovision/overall host load threshold and second- when deploying a new asset, you have a choice of which node to pick. Presumably your deployment tool (WHMCS, whatever) has visibility into your individual cluster.
2. If a running asset spikes to exceed a specific node's load threshold, you can live migrate it. again, you should be monitoring your cluster load in real time and can issue commands on trigger.
3. I totally get the desire to have a single tool you can buy for doing all of this- but in my experience the best solution is to write your own and use best of breed elements for all the middle pieces. PVE has a pretty broad api which allows pretty wide control over how the cluster behaves, and a well documented export mechanism for telemetry for your logic to trigger from.
4. While PVE has some RBAC functions, this should probably not be exposed to your tenants; your front end (which will also collect billing and usage metrics) should be the trigger point for customer interactions. It also give you arms length access control, since access is driven through a platform you have control over without ever handing keys to the hypervisor. Besides, there are a number of actions that require root access (eg asset deletions) which you dont want to expose.

If you're serious about operating a multi tenant environment for real, I'd urge you to obtain the talent and staffing to prevent.... missed customer expectations.

 

[Raph123FR]

One thing I would consider the availability of support. While the Proxmox Datacenter Manager (despite it's still limited feature set compared to vcenter) is covered by Proxmox support beginning with basic subscription level I'm not aware of any support offering for Pegaprox. On the other hand at least one of Pegaprox developers ( @gyptazy ) works at Proxmox Partner Credativ, maybe they are able to provide support? Asking them shouldn't hurt I guess
mutliportal has support included in their subscription, although their price might be a bit too much for your budget. They have custom offers for large scale deployments though according to their website: https://multiportal.io/pricing/

A couple of comments:

1. Since you're deploying client assets in a cluster, there are two components to consider- first, what is your target overprovision/overall host load threshold and second- when deploying a new asset, you have a choice of which node to pick. Presumably your deployment tool (WHMCS, whatever) has visibility into your individual cluster.
2. If a running asset spikes to exceed a specific node's load threshold, you can live migrate it. again, you should be monitoring your cluster load in real time and can issue commands on trigger.
3. I totally get the desire to have a single tool you can buy for doing all of this- but in my experience the best solution is to write your own and use best of breed elements for all the middle pieces. PVE has a pretty broad api which allows pretty wide control over how the cluster behaves, and a well documented export mechanism for telemetry for your logic to trigger from.
4. While PVE has some RBAC functions, this should probably not be exposed to your tenants; your front end (which will also collect billing and usage metrics) should be the trigger point for customer interactions. It also give you arms length access control, since access is driven through a platform you have control over without ever handing keys to the hypervisor. Besides, there are a number of actions that require root access (eg asset deletions) which you dont want to expose.

If you're serious about operating a multi tenant environment for real, I'd urge you to obtain the talent and staffing to prevent.... missed customer expectations.

Thank you for your feedback.

When I refer to “client,” I’m actually talking about internal users of the applications. Currently, certain (technical) staff members have the autonomy to manage snapshots, restart VMs, or perform other types of actions but in a limited scope.

For now, we’re still in the solution evaluation phase (XCP-NG is also under consideration, along with Hyper-V), but the other solutions either don’t align with the company’s strategy or require too significant technical overhaul.

Coming from VMware, this is a major change, and of course, we need to identify all potential issues at this stage ! And one of the most important issues I'm trying to resolve is this resource management ...

 

[Fearless-Grape5584]

Thank you for your feedback.

When I refer to “client,” I’m actually talking about internal users of the applications. Currently, certain (technical) staff members have the autonomy to manage snapshots, restart VMs, or perform other types of actions but in a limited scope.

For now, we’re still in the solution evaluation phase (XCP-NG is also under consideration, along with Hyper-V), but the other solutions either don’t align with the company’s strategy or require too significant technical overhaul.

Coming from VMware, this is a major change, and of course, we need to identify all potential issues at this stage ! And one of the most important issues I'm trying to resolve is this resource management ...

Looking for a product that already covers your requirements is the right first move. The catch is that for this particular combination I do not think one exists yet, so whatever is missing has to be built, and that is what you are trying to avoid. So the practical path is probably triaging the requirements: decide which limits really need to be hard, and accept soft limits or reporting only for the rest.

The disk part is a good example of why. When I looked into it, I could not find a way to get an accurate per-VM usage figure out of LVM-thin once snapshots share blocks with their origin, so any per-tenant total you compute will drift from what the pool actually consumes. I should be clear that I stopped there and did not evaluate the thin-provisioning-tools side, thin_ls in particular, so this is where my own investigation ended rather than a firm conclusion. If anyone here has gone further with it, I would be glad to be corrected.

ZFS looks like a better position for this. Usage attributable to snapshots is reported per dataset, and a dataset quota is enforced by the filesystem itself rather than by logic you have to maintain. LnxBil pointed at that approach earlier in this thread. If anyone were to build tenant level disk accounting, targeting ZFS only and dropping LVM-thin would probably make it a far more tractable problem.

CPU and memory are the easier half, since allocation can at least be checked at pre-start and pre-migrate. Actual disk usage, as opposed to provisioned size, is the one I would treat as reporting rather than enforcement.

Anyway, I am curious how your evaluation turns out. With XCP-NG and Hyper-V also in the mix that is an interesting comparison, and resource management does not look solved on any of them. Please do post back when you learn more.

 

[alexskysilk]

I could not find a way to get an accurate per-VM usage figure out of LVM-thin

ZFS looks like a better position for this.

Worth noting that neither of those storage options are valid for cluster use. If the scope is single host, just monitoring disk space ought to suffice. As far as I know, there is no storage solution that offers snapshot quotas regardless of hypervisor.