Ceph OSD and VM, CPU affinity: opinions and suggestions.

[Heracleos]

Hello everyone,
I would like your opinion on an idea that came to mind about a possible optimization or improvement in performance.

In short, do you think it makes sense to assign some CPUs specifically to OSDs and others to VMs, to avoid possible “collisions” between processes?

In my case, each node in my cluster has 48 cores (2 CPUs, 12 physical cores, 24 in hyperthreading for each CPU) and 10 HDDs, and therefore 10 OSDs (one each HDD).
I would like to assign CPUs 0-1 to the physical host's Debian, CPUs 2-12 to the OSDs only, and all other cores to the QEMU processes for the VMs.
This should be possible (I think) by modifying some scripts in systemd and adding an entry such as
[Service]
CPUAffinity=Nstart-Nend

Do you think this approach makes sense, or is it just a waste of time, because the physical host's OS already has an optimized balancing policy ?

 

[BobhWasatch]

Yes, the Linux kernel scheduler is total garbage and you need to manually tell it what to do. For "performance". Of the hardware I guess, since this scheme will hinder the performance of the admins having to maitain it.

 

[Heracleos]

Yes, the Linux kernel scheduler is total garbage and you need to manually tell it what to do. For "performance". Of the hardware I guess, since this scheme will hinder the performance of the admins having to maitain it.

Okay, so if I understand correctly, you believe that assigning CPUs to various processes makes sense.
It would be interesting to understand whether this feature is not directly available in the web GUI because the Proxmox developers haven't thought of it yet or because they simply consider it unnecessary.

 

[BobhWasatch]

I guess I should have put <sarcasm> tags on that. No, I don't think it makes any sense. It is a lot of effort for basically no return. It is a silly idea in the vast majority of cases. I'm not saying no one should ever do this, but it is ridiculous to think it should be the default or on the GUI.

 

[Heracleos]

... It is a lot of effort for basically no return....

Okay Sorry...
I didn't realize your answer was </sarcastic> .
Thank you very much for your thoughts. I think it's an excellent point of view.

 

[SteveITS]

The question/issue arises because https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster specifically says "As a simple rule of thumb, you should assign at least one CPU core (or thread) to each Ceph service..." Now whether that's necessary and worth the admin's time may depend on the disk speed (as noted) and maybe other factors like network speed (hence storage speed) and CPU usage in the cluster. I decided it wasn't.

 

[Heracleos]

The question/issue arises because https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster specifically says "As a simple rule of thumb, you should assign .....

The documentation mentions “to assign,” but in reality there is no way to explicitly assign a certain number of cores to certain processes.
Unless you do it manually by changing the settings on systemd or in other configuration files.
Maybe the documentation refers to ensuring that sufficient resources (in the case of CPUs) are left for the processes.