[SOLVED] Out of curiosity: where/how does CEPH store data?

[proxwolfe]

Hi,

I have a small PVE cluster in my home lab with CEPH. Each of the three nodes has an HDD and an NVME drive. All three HDDs form a pool and all three NVMEs also form a pool. The pool size is 3 (with a minimum of 2).

Originally, I thought that if I have a CEPH OSD in each node, all data would be replicated on every node and access to it would be just like on a local drive.

But now I am wondering whether that is actually correct or whether the data is spread across all OSDs and when I access a file it is put together from pieces stored on the local OSD as well as pieces on the other OSDs (on the other nodes) that have to be fetched via network.

Could someone please enlighten me?

Thanks!

 

[oz1cw7yymn]

Quick response.

For a replicated pool (size x), your original thought is correct. While files are placed in objects which are grouped in placement groups, rather than existing directly as files in a traditional file system, it is correct that all the data is copied to all the (x number of) osds in your scenario. Each data write to ceph needs to write the data to all the copies. Each read needs just one. Ceph can handle to lose x-1 disks without data loss, in your case loss of two disks. It will not allow you to use the data if below min-size number of disks.

For an erasure coding pool (x+y), data is split up into smaller chunks and distributed across x number of osds, with y number of parity chunks. So with erasure coding each data write needs to write to x+y disks/osds and read from x number of disks. Ceph can handle to lose y number of disks without data loss.

With just three nodes, you don't have a lot of flexibility. Replica of three is probably the best, but remember you're consuming three times the amount of data in storage usage.

 

[proxwolfe]

Thanks mate, very clear and concise - appreciate it.