Cpu cores and threads.

[Joseph Chrzempiec]

Hello I'm still new to proxmox but my question is when i create a vm how can i see how many threads i can use? Becuase i know i can select the cores but what about the threads. How do they work. My server is 12 cores 24 threads but i was curious for a windows vm.


Joseph

 

[apoc]

Threads don't count. You really should stick with the physical cores you have. They will help to "park idle cores" but if you really are doing something on the CPU, dont count on them...
And it is also a good practice to not "just assign everything". Your guest won't get faster using this approach, because it creates unnecessary overhead on your host.
Assign the CPUs you need.

• In case of a Windows Client OS - start with 2
• In case of a Windows Server OS - start with 2 (I'd even go to start with 1 - typically you have less "stuff" on it)
• In case of a Linux Server - start with 1. You likely will not even using that one up.

Of course that always depends on what you are doing. What I really want to emphasize is:
The design premise is to have "at least possible vCPUs" per VM.

 

[Joseph Chrzempiec]

Hello tburger thank you. I'm just learning this as i go along. One thing I'm wondering is that i know the server is 12 cores yet however in a vm creating it i can go past that. Wondering why is that. Is there no detection on how many cores the server has.

 

[apoc]

I'm just learning this as i go along.

No problem mate.

Technically you can assign as much cores as the underlying hardware has threads.
That does not make sense but it is technically possible.
(Haven't tried to go beyond that - as it makes even less than no sense )

You could argue "why is that" - the same argument one could have with driving without seatbealts. You can do it - but it doesn't make sense.
As always you should "know what you are doing" or at least have some basic understanding of things.

Imho you did the right thing: ask if something is unclear. Check out the web on "logical threads / cores / processors vs. physical cores / processors".

 

[Joseph Chrzempiec]

Thank you for the information. I learn a lot.

 

[PelleH]

I have a question in the same area.
If I have a Proxmox server with 2 CPU and 72 cores. I set up only one VM and give it 1 core - will the VM use only one core on the physical server?
I ask because I have some applications that are licensed per core.

 

[apoc]

Yes exactly. 1vCPU can only be scheduled to one core. So technically you can't use more than one core. Typically (without pinning) this is not always the same Core though.
Hint: Check the license agreement of your software. Sometimes there are tiny but nasty regulations about virtualized workloads...

 

[Octavian]

one more question @tburger please about the same:
statements:

1. 1vCPU = 1 Core
2. Stick with the physical cores you have(physical count cores - not hyper-threading)

the CPU sum of all running VMs should be <= number of physical cores? is that right?

 

[apoc]

1vCPU = 1 Core

That not exactly correct.
1vCPU is going to be scheduled to a logical core. A logical core can be a physical core or a hyperthread.
So technically a physical core with HT function provides two logical cores.
It is important to understand that while your system sees 2x the logical cores it actually gains a maximum of 20-30% additional performance. And that is only true if your workload matches (for instance if you have idle vCPUs (for instance waiting for IO to complete) that can be scheduled on a logical core which is backed by a HT-logical processor.

Stick with the physical cores you have(physical count cores - not hyper-threading)

That is the absolute maximum one VM should be assigned.
If all vCPUs of that one VM want to consume CPU time you otherwise paint the system into a corner which isnt good. Also remember that the host needs cycles too. This configuration is still very very bad. But you can make it worse by configuring a VM with a vCPU number of all logical cores.
Rule of thumb is to size against the minimum requirements. Every vCPU assigned needs to be scheduled. That means caches and registers need to be flushed etc. That creates an overhead. And I have seen it plenty of times that the scheduling wasted more cpu cycles than the actual application. Reducing the number of vCPUs drastically improved the performance on all areas.

the CPU sum of all running VMs should be <= number of physical cores? is that right?

It depends.
In general I would say no. Because it defeats the purpose of virtualizatio n.
Here comes the vCPU : pCPU ratio into place.
This ratio describes how many virtual CPUs have to share a physical CPU. important: threads (HT) don't count!
If you would do as you have mentioned then your ratio is 1:1. That is excellent from app-perspective.
Typically I see a 4:1 to 6:1 in the field.
But again this depends on two things
1. Your processor. When running low frequency CPUs with 2 GHZ you may only have a 4:1 while with a 3GHz you can do a 6:1
2. Your application. General Workloads can go with 4-6:1, but databases for instance often require a 2:1 or even 1:1. VDI desktops (depending on user profile) can work fine with 10-12:1.

So you see it all depends on the requirements.

 

[Octavian]

thanks @tburger for the details, very helpful.
If you don't mind, lets simulate a situation. Examples are very useful .
1 socket, 14 pCPU(28 vCPU) - 2.40 GHz.
Requirements: build a kubernetes cluster. lets say 3 master nodes and 4 workers.
a worker node will have, lets say a max of 20 dockers ... medium load app.
What are your recommendations for this situation
Thanks!

 

[apoc]

1 socket, 14 pCPU(28 vCPU) - 2.40 GHz.

Just to get the wording correct. 14 pCPUs and 28 logical CPUs / Threads
vCPUs are the CPUs assigned to / within the VM

What are your recommendations for this situation

This is really guesswork, because no-one knows what medium-load means.
But I am giving you an example:

Option 1: 2:1 vCPU to pCPU target ratio -> 28 vCPUs available
4x 4vCPUs for workers: 16 vCPUs
3x 2vCPUs for the masters: 6 vCPUs
SUM: 22 vCPUs is less than 28 vCPUs (2:1 ratio)

Option 2: 3:1 vCPU to pCPU target ratio -> 42 vCPUs available
4x 8vCPUs for workers: 32 vCPUs
3x 2vCPUs for the masters: 6 vCPUs
SUM 38 vCPUs (less than 42) - within ratio

I'd go with option 1 (smaller VMs) and rather scale the number of workers (scale out). Container-Workloads are typically perfectly engineered for this. This is "cloud native" - means scale out architecture. Maybe decrease number of containers per worker...
the bigger VMs are the more costly they are to schedule. So my expectation is that with 4 vCPUs per VM you run better than with 8vCPUs

Hope that makes sense.

 

[AllanM]

Hello I'm still new to proxmox but my question is when i create a vm how can i see how many threads i can use? Becuase i know i can select the cores but what about the threads. How do they work. My server is 12 cores 24 threads but i was curious for a windows vm.


Joseph

Hi Joseph,

Modern "big core" CPU architectures have very wide instruction pipelines that often have idle resources when working on a single thread. In order to make use of these idle resources, modern CPU core design has "doubled" the front-end of the CPU so it can schedule 2 threads on back-end compute resources simultaneously in an effort to keep them saturated. This is why we see "24 threads" presented to the operating system on a 12 core CPU.

The first "busy" thread scheduled to a given core can saturate 60-90% of the available compute throughput of that core depending on workload and core architecture. Once all cores have a busy thread, additional work/threads are scheduled in a "balanced" manner on both threads of each cpu, resulting in a more complete saturation of compute resources, which buys more total throughput, but a reduction in throughput available to any single thread, as would be expected.

If we allocate 12 "cores" to a VM, what we're actually doing is presenting 12 "threads" to the VM's operating system, which, on a host with a 12C/24T CPU, would allow the VM to saturate ~60-90% of available compute resources on the host under heavy workloads. If this is a good fit for the intended use, then that may be a good solution.

In datacenter or enterprise environments, the "right" allocation of CPU resources is a careful balancing act of making sure each VM has enough threads assigned that its own services run smooth are minimally impacted by other workloads within the VM, but not so much that it can negatively impact other VM's or the host.

Considerations:

Many virtualized servers can host their services with a single core assigned without much trouble, but the operating system may not be able to efficiently perform security/feature updates or virus scans or other maintenance activity on a single core very effectively. Remember to assign enough resources to perform the most intensive task at a reasonable level of efficiency. I would never assign less than 4 threads to a Windows VM, not because it needs it for whatever services it might be running, but because I want to be able to run updates on it quickly and have good interactive performance in the VM while it is busy.

Some types of VM's are very sensitive to having access to low latency, high throughput performance on the threads that are assigned. For example, a virtualized firewall that is handling a complex environment may have lots of virtual interfaces running IDS/IPS, DNS filtering, complex firewall rules, and many users connecting to multiple vpn instances/ports. This virtualized firewall might live on the same node with sensor VMs that monitor network bridge activity. In order to prevent such an environment from having high packet loss on sensors or causing poor DNS response performance or causing poor VPN performance for users, it's important that the allocation of resources on this type of scenario be carefully considered, enough threads for each VM to do its job but restrictive enough that the firewall always has low latency packet processing.

In a hyper-converged cluster, it's important that we're always leaving enough compute resources available to the hypervisor to have low latency access to the software defined storage that the node participates in hosting. An overburdened node could wind up negatively impact the disk access performance of an entire cluster. This doesn't mean that we can't "over-provision" resources, it just means we have to understand the workload and the likelihood of that over-provision ever actually resulting in a saturated CPU scenario. It's normal to assign several "cores" to a VM but have it sit mostly idle most of the time.

 

[Octavian]

Just to get the wording correct. 14 pCPUs and 28 logical CPUs / Threads
vCPUs are the CPUs assigned to / within the VM


This is really guesswork, because no-one knows what medium-load means.
But I am giving you an example:

Option 1: 2:1 vCPU to pCPU target ratio -> 28 vCPUs available
4x 4vCPUs for workers: 16 vCPUs
3x 2vCPUs for the masters: 6 vCPUs
SUM: 22 vCPUs is less than 28 vCPUs (2:1 ratio)

Option 2: 3:1 vCPU to pCPU target ratio -> 42 vCPUs available
4x 8vCPUs for workers: 32 vCPUs
3x 2vCPUs for the masters: 6 vCPUs
SUM 38 vCPUs (less than 42) - within ratio

I'd go with option 1 (smaller VMs) and rather scale the number of workers (scale out). Container-Workloads are typically perfectly engineered for this. This is "cloud native" - means scale out architecture. Maybe decrease number of containers per worker...
the bigger VMs are the more costly they are to schedule. So my expectation is that with 4 vCPUs per VM you run better than with 8vCPUs

Hope that makes sense.

Thanks @tburger

 

[wbravin]

Hello all

I am playing with my Proxmox (dell R710 1 cpu 6 cores and 32 gb ram) on and off for the past year in my home I will become my main media home automation server and main router .

I am pretty happy with my current configuration. I would like to improve it before i go live next week.
this is my current situation.

1. proxmox 7.4 installed
2. opnsense: installed as a vanilla VM, not active full time, when i connect it it is working as it should. It will eventually only act a a simple router
3. Home Assistant OS: Installed and configured correctly when active works fine. eventually it will be running constantly with Zwave, zigbee, BT and wifi connectivity via usb dongles. Little activity on a daily basis (motion detection Geofencing, Media environment Climate control
4. Windows 10: installed for learning purpose, worked fine now deleted
5. Truenas scale: Installed, not fully configured (missing HDD) not active

In reading all of the above, i still have a very very limited understanding on how to correct assign and manage CPU, Cores and ram to the vm. without being afraid of screw it all up (i can do this on my own without trying)

I would turn this serve on once a week to try different things.

Next week I will receive a proper HBA to use with Truenas and the necessary HDDs that i will need to learn how to pass them through to truenas. This will be an adventure.

at the moment i have assigned 2 cores to each vm.

In bringing online Truenas (once configured hopefully next week) will I need to allocate to truenas more cores and ram?

Turemas will have 20tb of drives. Therefore (according to ixsystems it should be allocated at least 28 GB of ram. (i will only use truenas on this server to feed music and videos to my pcs. and once a week to perform a full backup to another truenas server

Once allocated cores ram and cpu to a VM are they carved in stone or will proxmox manage these resources as required by the vm to the maximum allocated resources of the VM?

Therefore can i presume that opnsense and HA although allocated 2 core will only be actually using 1 leaving 4 for truenas?

If I add a second cpu with 6 more cores and an additional 32gb ram can i allocate 1 cpu to OPNsense and HA and one cpu to truenas using all 6 cores of the new CPU?

do i really need to do this? or should i just add ram to the current cpu?

In the same line, If i add a second CPU will i need to pass-through the USB separately to each vm on each cpu?

Thank you in advance for your patience and help?

 

[Dunuin]

In bringing online Truenas (once configured hopefully next week) will I need to allocate to truenas more cores and ram?

ZFS needs alot of RAM and CPU performance. I for example gave my TrueNAS Core VM with 4x 8TB HDDs 32GB RAM + 8 vCPUs and when writing stuff I can see the CPU peaking at 100% utilization (but I'm also using encryption and zstd compression streasing the CPU a lot). So yes, 2 vCPUs isn't great. I would add 4 vCPUs to both the TrueNAS as well as the Window10 VM. Here the Win10VMs run best with 6 vCPUs + 8GB RAM. With 4 vCPUs and 6GB RAM browsing was a bit unresponsive.

Once allocated cores ram and cpu to a VM are they carved in stone or will proxmox manage these resources as required by the vm to the maximum allocated resources of the VM?

With PCI passthrough of the HBA the VM will always use the full RAM, no matter how much TrueNAS is actually using. So assign it 28GB RAM and it will always use the full 28GB + some overhead (so probably more like 29-30GB).

In difference to the CPU ressources you can't really overprovisioning your RAM. So yes, 32GB RAM is very low for your workload.

 

[wbravin]

ZFS needs alot of RAM and CPU performance. I for example gave my TrueNAS Core VM with 4x 8TB HDDs 32GB RAM + 8 vCPUs and when writing stuff I can see the CPU peaking at 100% utilization. So yes, 2 vCPUs isn't great. I would add 4 vCPUs to both the TrueNAS as well as the Window10 VM. Here the Win10VMs run best with 6 vCPUs + 8GB RAM. With 4 vCPUs and 6GB RAM browsing was a bit unresponsive.



With PCI passthrough of the HBA the VM will always use the full RAM, no matter how much TrueNAS is actually using. So assign it 28GB RAM and it will always use the full 28GB + some overhead (so probably more like 29-30GB).

In difference to the CPU ressources you can't really overprovisioning your RAM. So yes, 32GB RAM is very low for your workload.

First many thank you your prompt reply.

So As far i can understand, on this server with this cpu i will definitely need to increase the ram to 64 therefore 32 for truenas and 16 for each of the other vm (i will not run windows or debian as vm on this server)

If on the other the hand if I install a second 6 core CPU i can add the additional 32 gb ram to this CPU This should be comfortable
or
I could dedicate

1 cpu with 6 cores and 32 gb ram to truenas and
1cpu with 6 cores and 32 GB ram to OPNsense (2 cores) and Home assistant (2 cores)

This too would be comfortable

which would be the most efficient and easiest to manage in your opinion?

Again thank you for your time and assistance

 

[apoc]

So As far i can understand, on this server with this cpu i will definitely need to increase the ram to 64 therefore 32 for truenas and 16 for each of the other vm (i will not run windows or debian as vm on this server)

*eek* - no no no. Don't forget the hypervisor host.
At least spare 4 GB for PVE itself and you shouldn't assign 100% anyways.

1 cpu with 6 cores and 32 gb ram to truenas and
1cpu with 6 cores and 32 GB ram to OPNsense (2 cores) and Home assistant (2 cores)

This is not how virtualization works. This whole thread discusses the mechanisms of it.
But with the numbers mentionend you should be fine.

I'd take another approach though. Stick with current config and find the bottlenecks. Then adjust.

 

[wbravin]

Thank you for this heads up, your teaching and patience

Based upon this discussion I am thinking of reducing the cores to one for opnsense and Home Assistant and allocating 4 cores to truenas.
in addition i allocated
2 gb to opensense
2 gb to HA
24 gb to truenass

This leads me to think that i would need to increase the ram to at least 64gb.

My current server will not support 8 core cpu. and before adding the ram.

I think i could consider the option to sell the 710 and move everything to a barebone 720 gen 2 o 730. here i can find barebone 720 servers already equipped with an 8 core cpu (xeon 2650) 64 gb or 128 gb ram that i can by for about 300 euros

As i made a fair investment (from my perspective) on the dell 710 , In this case i can move hba, caddies and disks to the 720

In this case

I would have

1. opnsense 2 cores and 8gb ram. This vm really does not need any more than this
2. Home assistant 2 cores and 8 gb ram. This vm does not rally need more than this
3. truenass 4 cores and 32 gb ram. My truenass will have more than it requires

None of the hardware requirements pages for these softwares state more than these parameters

leaving a minimum of 16GB free for pve

in this case all vms are happy and so will pve

However if i understand correctly if i have 2 cpus with 8 cores each, then i could set up each vm with 2 cpus with the number of cores required up to a total of 16. Correct?

I will however increase increase the ram to 96 or 128 just to be safe.

Therefore a r720 with a xeon 2650 and 128 gb should work fine with no issues (unless i create them) correct

Again thank you for your patience and teachings

be well

 

[wbravin]

Hello all

I have my proxmox 8.0.4 up and running fine. thank you all

I installed 128gb ram on the server thinking that this would be enough for the vms to "swim" freely.

Just to tweak and nitpick

My issue
When i need to reboot the proxmox server (installed on a dell r720 with 2 X 5-2660 and 128gb ram) the boot process and the start of the vms seams slow

I allocated 4 cores to truenas core and 80 gb ram.

It still takes 0ver a minute to start and i see that the truenas vm is using 99% of the available ram and at some points 90 % of the cores.

Since I only have 24 TB of spinning drives and the boot drive is an 480 mb ssd I would imagine that this process would be faster.

As a precaution and practice (maybe not necessary) When either proxmox or any of the VMs (or their components) have changes, upgrades and / or updates I tend to reboot the vm and since I am there reboot proxmox.

To speed this up (or am i being just impatient) what you i upgrade, the cpu's the ram or both, or should i just reset my expectation and let the vms play out as they are doing now?

Best practice
Do i need to start the vm sequentially or automatically? does it make a difference overall?

I do not want to start tweaking the system on my own because i'm certain i will foobar it

Thank you for your time, patience and help in this matter

 

[apoc]

When i need to reboot the proxmox server (installed on a dell r720 with 2 X 5-2660 and 128gb ram) the boot process and the start of the vms seams slow

What exactly is slow and how much time does each individual step take?

It still takes 0ver a minute to start and i see that the truenas vm is using 99% of the available ram and at some points 90 % of the cores.

To boot the vm? I think this is absolutely reasonable.

Best practice
Do i need to start the vm sequentially or automatically? does it make a difference overall?

I do boot my VMs partially sequential, partiall parallel. In the end it takes some time to get them all up. I have never tried to do everything in parallel. it also does not make sense because of dependencies...

Once we have some information about your exact times I think we can judge better.