[SOLVED] Specific Question on Ceph and OSDs from Compute Hosts

Seed

Active Member
Oct 18, 2019
98
62
38
124
I have been tooling around quite a bit with Ceph and Proxmox the last few weeks, and really, it is pretty awesome. It's really taken my lab of a mix mash of machines and hosts, and made them somewhat valuable again and much easier to manage my needs and experiments. I mean it's another level of enjoyment and ease of use where I used to have dedicated hardware for things strewn about.

I do have a question though around how Ceph works with disks and OSDs.

I have the following 5 hosts

Hosts 1 2 and 3 have:
  • 4 HDD OSD
  • 3 NVMe OSDs
  • 1 SSD OSD

Hosts 3 and 4
  • 0 OSDs

All hosts have 10Gb ethernet for storage and 10Gb for VMs/Management

I have 3 pools of the mentioned OSDs, that are available to ALL hosts
  • pool-ssd
  • pool-nvme
  • pool-hdd

So if a VM resides on HOST 1 for example, even though the disks are local, but the VM is installed into the pool-ssd which is the shared group, it will still run over network right? Meaning there really isn't any direct access to pools even if the host member has OSDs that are part of it.

If that is true, then HOST 4 or 5 will have the same level of performance with the VM even though it has 0 OSDs, but is allowed to run VMs because of the pools.

If that is true, then I would think the higher core clock speed of HOST 4, 3.5GHz, vs 2.1GHz will have better performance for single threaded tasks if both hosts access VMs from the same pool. True?

C
 
So if a VM resides on HOST 1 for example, even though the disks are local, but the VM is installed into the pool-ssd which is the shared group, it will still run over network right? Meaning there really isn't any direct access to pools even if the host member has OSDs that are part of it.
Ceph doesn't know locality. As a distributed network storage, it will always write data across the network.

If that is true, then HOST 4 or 5 will have the same level of performance with the VM even though it has 0 OSDs, but is allowed to run VMs because of the pools.
They merely act as clients.

If that is true, then I would think the higher core clock speed of HOST 4, 3.5GHz, vs 2.1GHz will have better performance for single threaded tasks if both hosts access VMs from the same pool. True?
This is true in any case, higher clock speed equals lower latency.
 
  • Like
Reactions: Seed

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!