How to allocate CPU resources for a VM correctly?

[LnxBil]

The CPU limit parameter limits the maximum usage of the hypervisor's physical cores? or are processor threads also taken into account?

AFAIK, cores and threads are indistinguishable from a process perspective, due to the abstraction inside of the Linux kernel and therefore you don't see any indication in tools like top and such. There are things at play that have a much higher impact like e.g. NUMA-locality.

If my processor has 4 cores and 4 threads, then theoretically I can put CPU limit =8?

Yes, but that's like setting no limit in this case according to the documentation.

As with hotplugging ... I don't get why I would need to set the limit in the first place, just give the VM less or more resources and scale horizontally.
Setting 16 cores and limiting them to the power of 4 has no benefits for me. Just give the VM 4 of them. It's the same as hotplugging only on another level. Having 16 cores with its increased memory consumption for scheduler, datastructures and such only to limit its power is a total waste of resources to restricting it beforehand. You may temporarily restrict them with the limited, but not in the beginning. In a perfect would, this would be dynamic like with QoS on the network side, yet that is currently not possible and AFAIK also currently not implemented fully in Linux as well. You would also want to have dynamic I/O limitations in place which are IMHO more important. I seldomly see memory or cpu limitions, just I/O.

 

[emptness]

AFAIK, cores and threads are indistinguishable from a process perspective, due to the abstraction inside of the Linux kernel and therefore you don't see any indication in tools like top and such. There are things at play that have a much higher impact like e.g. NUMA-locality.


Yes, but that's like setting no limit in this case according to the documentation.

As with hotplugging ... I don't get why I would need to set the limit in the first place, just give the VM less or more resources and scale horizontally.
Setting 16 cores and limiting them to the power of 4 has no benefits for me. Just give the VM 4 of them. It's the same as hotplugging only on another level. Having 16 cores with its increased memory consumption for scheduler, datastructures and such only to limit its power is a total waste of resources to restricting it beforehand. You may temporarily restrict them with the limited, but not in the beginning. In a perfect would, this would be dynamic like with QoS on the network side, yet that is currently not possible and AFAIK also currently not implemented fully in Linux as well. You would also want to have dynamic I/O limitations in place which are IMHO more important. I seldomly see memory or cpu limitions, just I/O.

You are saying everything correctly.
But we don't have 1 hypervisor and 3 virtual machines. We have a lot of them (102) and there will be even more. And many of them do not constantly use resources at 100%. Resources are idle.
I am considering a theoretical possibility for optimal settings so that we can deploy more VMs and not be tied to the number of physical cores. So that the hypervisor knows how to balance resources and if 1 VM does not use them, and the other does not have enough of them, then give them away.
In this case, we will save the organization money, we will not have to buy more servers.

 

[LnxBil]

But we don't have 1 hypervisor and 3 virtual machines. We have a lot of them (102) and there will be even more. And many of them do not constantly use resources at 100%. Resources are idle.

Yes, we have hundreds and of course the resources are idle most of the time. That's why we virtualize .. consolidation.

I am considering a theoretical possibility for optimal settings so that we can deploy more VMs and not be tied to the number of physical cores.

Why would you be tied to the number of cores? You always overprovision them, because not every VM needs it at the same time. More importantly, there is NO "optimal setting". This is always in a constant flow as I tried to explain to you. You cannot set only some cpu limited and number of cores, you also need to balance the workload over all cluster nodes and live migration machines around to get the overall utilization in a workable flow. As already said, there is currently no software that can do this, because it depends on a lot of different things that you can optimize or their own, but not combined: RAM, I/O, CPU, network. So basically every virtualized resource.

So that the hypervisor knows how to balance resources and if 1 VM does not use them, and the other does not have enough of them, then give them away.

Sure, that's how it works. Everything in PVE (more precisely in KVM) is a process with at least one thread that gets scheduled by the linux scheduler and you monitor if you use less than 100% of your overall CPU resources. The same is true for RAM, whereas you also have kernel samepage merging to pack more similar machines on your hardware. RAM is a bit different, because "free ram" does not exist, never has, never will and most of people still think this is not true (just look at all the posts in the forum about RAM usage beeing so high). It is always used for caching, so from a hypervisor perspective, it is always used and cannot be distinguished into "really used" and "just used for caching". You can mitigate this a bit with using ballooning that ONLY reduced the amount of your VM if it allows it and if another machine needs it. This is often misunderstood that it only uses what it needs, but that is wrong.

All the things we talked about is basic hypervisor 101, true for decades and implemented (more or less) in any hypervisor I've ever seen.