I am trying to restrict a group of users to specific resources on a specific host in our cluster. I have used a resource pool to limit VM and storage access and other permissions to restrict network access. This seems to work well except that they can create VMs on other hosts in the cluster. Those VMs don't have access to any resources - storage, networking, etc. - but I'd still like to prevent them from being able to do this. Is this possible?
AFAIK, that's not possible, because a cluster should have all the resources shares, at least in all the cluster I know is this the case.
For a more restrict setup, I can recommend to nest PVE inside of PVE and give all the resources you want to give to the group of users to the VM (maybe a little bit more for PVE itself) and give then access rights there. This is the only way to restrict all resources and cage the group of users. If you run e.g. ceph in your cluster, you can just create a new ceph pool and setup the nested PVE as a client so that you'll store the data directly in your main clusters ceph cluster without any more performance penalties due to the 2-level virtualization.
This seems like a novel approach to limiting access, but I think it would work. Performance isn't a primary concern for us here either. I'd be curious to know the implications on the network side of things, but I can play around with that. In any case, thank you!