[SOLVED] Specific Question on Ceph and OSDs from Compute Hosts

[Seed]

I have been tooling around quite a bit with Ceph and Proxmox the last few weeks, and really, it is pretty awesome. It's really taken my lab of a mix mash of machines and hosts, and made them somewhat valuable again and much easier to manage my needs and experiments. I mean it's another level of enjoyment and ease of use where I used to have dedicated hardware for things strewn about.

I do have a question though around how Ceph works with disks and OSDs.

I have the following 5 hosts

Hosts 1 2 and 3 have:

• 4 HDD OSD
• 3 NVMe OSDs
• 1 SSD OSD

Hosts 3 and 4

• 0 OSDs

All hosts have 10Gb ethernet for storage and 10Gb for VMs/Management

I have 3 pools of the mentioned OSDs, that are available to ALL hosts

• pool-ssd
• pool-nvme
• pool-hdd

So if a VM resides on HOST 1 for example, even though the disks are local, but the VM is installed into the pool-ssd which is the shared group, it will still run over network right? Meaning there really isn't any direct access to pools even if the host member has OSDs that are part of it.

If that is true, then HOST 4 or 5 will have the same level of performance with the VM even though it has 0 OSDs, but is allowed to run VMs because of the pools.

If that is true, then I would think the higher core clock speed of HOST 4, 3.5GHz, vs 2.1GHz will have better performance for single threaded tasks if both hosts access VMs from the same pool. True?

C

 

[Alwin]

So if a VM resides on HOST 1 for example, even though the disks are local, but the VM is installed into the pool-ssd which is the shared group, it will still run over network right? Meaning there really isn't any direct access to pools even if the host member has OSDs that are part of it.

Ceph doesn't know locality. As a distributed network storage, it will always write data across the network.

If that is true, then HOST 4 or 5 will have the same level of performance with the VM even though it has 0 OSDs, but is allowed to run VMs because of the pools.

They merely act as clients.

If that is true, then I would think the higher core clock speed of HOST 4, 3.5GHz, vs 2.1GHz will have better performance for single threaded tasks if both hosts access VMs from the same pool. True?

This is true in any case, higher clock speed equals lower latency.