Understanding Ceph Fundamentals

[Donovan Hoare]

Good Day All. I have just set up a 4 node cluster with Ceph.
So a few Questions to help me understand what is happening

1. I know data is stored over multiple servers. However, when I run a VM is the "Hard Drive" on the local machine copy for IO.
Or does the VM have to read and write over the network? I know when it writes it makes a copy over the network.

2. On one server i added 2 800GB SSD i partinioned them in 105GB sections.
Then added the SAS OSD by selecting an SSD partition for the DB.
So it's 6 SAS Drives to one SSD.

So my understanding here is that writes are written to the SSD and then moved to the SAS drives later.
Is my understanding correct? If not can someone please explain?
Does this also help at all with the read speed and bootup speed of a VM?

3. For the DB Disk, i created a partition for each disk.
However, when creating the OSD it kept the disks as an option.
Is a separate partition better or can you have one big partition and use it for all 6 OSDs

 

[sterzy]

I know data is stored over multiple servers. However, when I run a VM is the "Hard Drive" on the local machine copy for IO.
Or does the VM have to read and write over the network? I know when it writes it makes a copy over the network.

Well that depends on how you configured your VM. If you use Caching, not everything needs to be sent over the network at all times. However, yes, there is not 1:1 mapping between VM data, host, and disk. So you can't read directly from a Ceph disk that is located on your node. So you will have to go through the network layer for that.

So my understanding here is that writes are written to the SSD and then moved to the SAS drives later.
Is my understanding correct? If not can someone please explain?

Kind of, but mostly this applies to metadata. The DB only handles metadata and not the data itself. So data will still be written directly to the SAS drives.

3. For the DB Disk, i created a partition for each disk.
However, when creating the OSD it kept the disks as an option.
Is a separate partition better or can you have one big partition and use it for all 6 OSDs

No creating separate partitions is preferred if you really want to do this. However, note that if your SSD fails, all connected OSDs will also fail. So this will take down an entire node. So it would be better if you could use multiple SSD per node as DB disks.

Also note, that using SSDs as DB disks to speed up SAS HDDs is kind of a measure of last resort. You won't see great performance with such a setup, and we generally recommend that you use only enterprise SSDs.

 

[Nemesiz]

You can create crush rules to make localized reads