The link is not for QEMU/KVM and some generic boilerplate that isn't helping at all for the distinction of vCPU's and CPU units (a scheduler knob) that the original post asked about, or what does this link answers?
[SOLVED] Sockets vs Cores vs Threads vs vCPU vs CPU Units
- Thread starter ahriman
- Start date
-
- Tags
- cpu sockets threads vcpu
Thanks, it asks the sames questions that I posed. However, in the age of generative AI, I can't help but find the maths is a little questionable
(8 Cores x 16 Threads) x 1 CPU = 128 vCPUs
How would KVM be able to schedule all 100 vCPUs at the same time when there are only 16 actual threads? That's why the number of virtual cores (per socket) is limited to the number of physical threads. If you want your VMs to run smoothly, you should run your VMs with 8 virtual cores or less.
EDIT: I learned that this is not a KVM limitation but a reasonable PVE one.
Last edited:
Actually, the maximum number of vCPUs could be higher for KVM (NR_MAX_CPUs, i.e., what the kernel would support as maximum CPU count for the actual physical CPUs its running on).
But yes, it makes no sense whatsoever to have more vCPUs than there are CPU threads available, with that one is just making the scheduler run in cycles and waste CPU power on cycling through the extra cores.
That's why we block such over-allocation explicitly on a Proxmox VE management level, as this makes performance a lot worse, and if unnoticed it might be harder to debug for users.
Depending on what the host itself has to do, or if there are other guests, a smaller vCPU makes sense to avoid all to high CPU pressure overall.
But yes, it makes no sense whatsoever to have more vCPUs than there are CPU threads available, with that one is just making the scheduler run in cycles and waste CPU power on cycling through the extra cores.
That's why we block such over-allocation explicitly on a Proxmox VE management level, as this makes performance a lot worse, and if unnoticed it might be harder to debug for users.
Depending on what the host itself has to do, or if there are other guests, a smaller vCPU makes sense to avoid all to high CPU pressure overall.
AFAIK, for the kernel to schedule a VM to run, it has to wait until the number of vCPU configured in the VM are free, so the host's kernel can schedule all to run at the same time. So for a 4 vCPU VM to run, there has to be 4 hosts CPUs free, otherwise the kernel has to wait to schedule that VM execution. The wait may be very short, of course. This is why I set vCPU a low as possible for all VMs in a host.
Am I right?
for KVM/Qemu, vCPUs are "just" threads that are scheduled however the kernel is configured. you can optimize performance by limiting/avoiding overcommiting, pinning vCPU threads, adjusting scheduler and NUMA-related settings, .. and of course, there are certain optimizations in the code as well (in the guest kernel and Qemu) to avoid deadlocking by inter-vcpu dependencies (or at least reduce the chance of triggering one, unless you are gravely overcommitted).
I'm late to this thread and don't understand fully, I'm not very technical :/
I have the same issue i think:
Let's say I have 5 vms. Just say I have 2 cores and 8 threads total (4 threads per core).
Iv have tried to explain with pictures (and I know I forgot to put in proxmox requirements just forget them for a bit) how would I allow my "some docker thing vm" to use the rest of the server resources manually (NOT AUTOSCALE)? Would I just change the core count to 3 from 1? Thanks for any answers
(i just want to increase the resources that docker has to boost it performance for a bit)
I have the same issue i think:
Let's say I have 5 vms. Just say I have 2 cores and 8 threads total (4 threads per core).
Iv have tried to explain with pictures (and I know I forgot to put in proxmox requirements just forget them for a bit) how would I allow my "some docker thing vm" to use the rest of the server resources manually (NOT AUTOSCALE)? Would I just change the core count to 3 from 1? Thanks for any answers
(i just want to increase the resources that docker has to boost it performance for a bit)
Yep, probablyvery exotic. Dont know much about this tuff srry. And no, it's not. Its 4 socket 18 core per socket and 2 thread per core. But I'm trying to keep specs simple. The core and the thread count doesn't really matter I don't think.
Last edited:
When Proxmox boot, how much memory does it need - depends on what services you have running, with Ceph, GPU, monitoring etc, I’m at 1G of RAM a few seconds after boot without any VM. CPU if it’s not doing anything, it doesn’t need anything. The management web thingy can be run in a single thread, if you start talking live migration, heavy Ceph usage, network management etc, you could want more. Generally I’m overcommitting my CPU significantly and I have no problems because all VMs don’t ever simultaneously spin up, but when using Ceph to NVMe over 100G, it could max out a few cores.
You can simply not assign a set of RAM for Proxmox, the Linux kernel OOM killer will generally not kill any of the Proxmox processes because they are the longest lived and don’t use much memory compared to say, a VM. So generally your VM will be killed first if you’re overcommitted significantly.
You can simply not assign a set of RAM for Proxmox, the Linux kernel OOM killer will generally not kill any of the Proxmox processes because they are the longest lived and don’t use much memory compared to say, a VM. So generally your VM will be killed first if you’re overcommitted significantly.
Agree. There is no other answer than "it depends".
Pretty much the whole IO-profile plays a significant role.
Including the dependency of your hardware-setup (DMA-Access, Interrupts, etc.).
In my opinion you should primarily have a look on the CPU Oversubscription and memory assignment. I´d recommend to spare 2-4 GB for your PVE host.
When you have ZFS in place please don´t forget the pressure which ARC might bring to the system.
Pretty much the whole IO-profile plays a significant role.
Including the dependency of your hardware-setup (DMA-Access, Interrupts, etc.).
In my opinion you should primarily have a look on the CPU Oversubscription and memory assignment. I´d recommend to spare 2-4 GB for your PVE host.
When you have ZFS in place please don´t forget the pressure which ARC might bring to the system.
I appreciate this thread. Especially the thoughtful (and patient) responses from Proxmox staff.
My use case is not so much about hosting multiple VMs. I only run one VM at a time.* I want to reserve an appropriate amount of horsepower and memory for the host, then maximize the performance of my one VM. How to tune for that, is what led me to this discussion.
My hardware:
Ryzen 9 7945HX 16c/32t
96GB RAM
My VM 'hardware' config:
Processors 12 [host]
80GB RAM
Running Linux, if that matters
At the risk of contributing more confusion, here's what I understand after reading this discussion:
-Mark
* I think this inappropriate use case is not all that uncommon. I want to maximize the performance of my only VM. I give-up the bare-metal performance for the convenience of snapshot rollbacks. I'm constantly installing bleeding edge drivers and other unproven software. If something breaks, I can roll back to a prior snapshot. Proxmox has saved the day more than once.
My use case is not so much about hosting multiple VMs. I only run one VM at a time.* I want to reserve an appropriate amount of horsepower and memory for the host, then maximize the performance of my one VM. How to tune for that, is what led me to this discussion.
My hardware:
Ryzen 9 7945HX 16c/32t
96GB RAM
My VM 'hardware' config:
Processors 12 [host]
80GB RAM
Running Linux, if that matters
At the risk of contributing more confusion, here's what I understand after reading this discussion:
- I could increase the number of vCPUs on my VM, but I might only see a modest performance bump.
- I'm not suffering a CPU bottleneck now, so I might not bother trying until I do.
- I could experiment and benchmark, to find what works best for my workload.
- Benchmark actual throughput, as tools like htop might not tell the whole story.
-Mark
* I think this inappropriate use case is not all that uncommon. I want to maximize the performance of my only VM. I give-up the bare-metal performance for the convenience of snapshot rollbacks. I'm constantly installing bleeding edge drivers and other unproven software. If something breaks, I can roll back to a prior snapshot. Proxmox has saved the day more than once.
It is definitely a valid question, if all you're doing is a single host on a single VM, the overhead would be very minimal, you'd need probably ~1GB of RAM and 0.5 a core. The "problem" with hyper-converged infrastructure is that there isn't a clear delineation between your network, storage, security and virtualization 'overhead', so you have to compose and scale them according to particular workload needs, which allows for very cost effective, dense rack configurations, but as you scale, you need to put at least some question, unlike other platforms like VMware that recommend (aka require if you want a supported solution) to buy two racks of storage and a rack of compute and a half rack of networking to get what can be done in a single rack of hyperconverged (and yes, VMware tried their hand at hyperconvergence, it's a disaster area).
As far as your Ryzen, it's desktop-grade, and Zen as a platform is more limited by its own design (the NUMA complexities of low memory bandwidth chiplets etc). In some cases (eg. low threading, high memory bandwidth applications), adding more CPU threads may actually slow down the system - it's not for nothing Intel still sells "just" an 8-core Xeon clocking at 4.2GHz.
So the same question arises, what are you doing with your VM that would demand adding more CPU, you may have to benchmark and try out some different configurations using tools like numastat to figure out what your 'ideal' configuration is. Eg. if you're just adding 2 cores from an 8 core chiplet, it may not be useful enough to do that as anything scheduled on those CPU will be slower than the other 8.
As far as your Ryzen, it's desktop-grade, and Zen as a platform is more limited by its own design (the NUMA complexities of low memory bandwidth chiplets etc). In some cases (eg. low threading, high memory bandwidth applications), adding more CPU threads may actually slow down the system - it's not for nothing Intel still sells "just" an 8-core Xeon clocking at 4.2GHz.
So the same question arises, what are you doing with your VM that would demand adding more CPU, you may have to benchmark and try out some different configurations using tools like numastat to figure out what your 'ideal' configuration is. Eg. if you're just adding 2 cores from an 8 core chiplet, it may not be useful enough to do that as anything scheduled on those CPU will be slower than the other 8.
Last edited:
@marked23 : In my humble opinion the "issue" starts at a different place.
The "one-vm-on-the-hypvervisor" setup you describe is an edge-case. Hypervisors were built to have massive-parallel work to do, because all the time some of the VMs is doing a little. But not all VMs need everything the same time.
Once you get into the area/situation that all CPU-Power needs to go to one VM you are better of with a bare metal installation - at least from the performance perspective. This means you need to accept some tradeoffs performance-wise if you consider running this workload/VM virtualized.
The main reason for this tradeoff is the fact that your guest (e.g. the OS in the VM) does not have any visible indicator which thread is a physical core and which is not. Or to put it differently: For your VM every CPU-core, which you assign to it, looks the same. The Guest-OS therefore does not know that basically every second thread is going to the same physical core of your Ryzen-Processor.
This also means that if you assign more virtual CPUs to the VM you put the Hypervisor into an "unsolvable" situation. It simply can´t match the requests once the VM does "need" all CPU-cores.
From my perspective this will lead to a degredation of the performance as soon as you need all cores in the VM.
Even though you might have the "feeling" that you are missing out on a lot of potential performance in my opinion you are not.
Because Multi-Threading from my experience does not give you a ton of extra-performance. It helps you in certain scenarios. E.g. if a thread waits for an underlying hardware-device (IO-Wait). But aside that it wont double your available CPU-performance, because simply there only is one physical CPU/Core underneath the 2 threads it offers.
You can only gain a benefit from Multi-Threading if the OS is aware of the fact.
But from all what I know PVE / KVM can not provide that information to the guest-OS: Hence your guest-OS also can´t make proper use of these "extra-capabilities".
For your "special situation" I think the most CPU you should give to your VM is 14. This would leave your host 2 addiitonal cores to work with without interfering with the VM directly.
Everything else would not make sense to me.
I might be wrong though - you would need to find out by experimenting ...
All the best. Please let us know how it turnd out
The "one-vm-on-the-hypvervisor" setup you describe is an edge-case. Hypervisors were built to have massive-parallel work to do, because all the time some of the VMs is doing a little. But not all VMs need everything the same time.
Once you get into the area/situation that all CPU-Power needs to go to one VM you are better of with a bare metal installation - at least from the performance perspective. This means you need to accept some tradeoffs performance-wise if you consider running this workload/VM virtualized.
The main reason for this tradeoff is the fact that your guest (e.g. the OS in the VM) does not have any visible indicator which thread is a physical core and which is not. Or to put it differently: For your VM every CPU-core, which you assign to it, looks the same. The Guest-OS therefore does not know that basically every second thread is going to the same physical core of your Ryzen-Processor.
This also means that if you assign more virtual CPUs to the VM you put the Hypervisor into an "unsolvable" situation. It simply can´t match the requests once the VM does "need" all CPU-cores.
From my perspective this will lead to a degredation of the performance as soon as you need all cores in the VM.
Even though you might have the "feeling" that you are missing out on a lot of potential performance in my opinion you are not.
Because Multi-Threading from my experience does not give you a ton of extra-performance. It helps you in certain scenarios. E.g. if a thread waits for an underlying hardware-device (IO-Wait). But aside that it wont double your available CPU-performance, because simply there only is one physical CPU/Core underneath the 2 threads it offers.
You can only gain a benefit from Multi-Threading if the OS is aware of the fact.
But from all what I know PVE / KVM can not provide that information to the guest-OS: Hence your guest-OS also can´t make proper use of these "extra-capabilities".
For your "special situation" I think the most CPU you should give to your VM is 14. This would leave your host 2 addiitonal cores to work with without interfering with the VM directly.
Everything else would not make sense to me.
I might be wrong though - you would need to find out by experimenting ...
All the best. Please let us know how it turnd out
Last edited: