CEPH capacity?

[robertut]

My homelab Proxmox test environment is made of 3 nodes in a cluster. I have installed in each a dedicated 256GB NVMe disk to test and learn CEPH.
(note: the video tutorial on https://www.proxmox.com/en/training/video-tutorials/item/install-ceph-server-on-proxmox-ve is heavily outdated, many things don't apply like how it's presented there, I suggest to at least add a note about which Proxmox version that video applies to).

I created everything with the default settings. Every NVMe disk is an OSD.

My question is how to estimate the space budget in the storage pool?

In the status page it shows that the available storage space 715.42 GiB which is roughly the sum of all the disks. Shouldn't the available space shown be much smaller? I thought that since the data is replicated across all the disks, the space should be occupied 3 times, thus the total/available space should be a third of that isn't it?
And how about overprovisioning?

 

[aaron]

With Ceph you need to distinct where you look at available space. In the Ceph panels you usually see the raw storage capacity.

Once you have a pool with the matching storage config in Proxmox VE, you will see the estimated usable space in the summary of the storage.

A lot of factors play into how much of the raw capacity can be used. For example a few are:

• size parameter of the replicated pool(s)
  • with a size 2 and 3 nodes, you get more usable space for example as only 2 replicas are stored in the cluster
• EC pools are even more different
• fullest OSD limits how much free space is estimated
• ...

 

[robertut]

Thank you. The raw capacity in the status page of CEPH mislead me.

Can I overprovision on the CEPH storage? Eg. use a 300GB virtual disk holding only 40GB of data (rest is empty) on a 243.05 GB CEPH storage?

 

[aaron]

Can I overprovision on the CEPH storage? Eg. use a 300GB virtual disk holding only 40GB of data (rest is empty) on a 243.05 GB CEPH storage?

Yes. On Ceph RBD we cannot reserve any space. While it would be possible to zero out a newly created disk image, that would be gone with the first trim.
Make sure to enable the "discard" option for your VM disks so that a trim/discard is passed down to the RBD layer to free up unused space. And keep an eye on the free space. Ceph can handle a lot, but running out of space is one of the more painful problems to get out of again.

 

[robertut]

What's your suggestion to test CEPH high availability/redundancy?

I have set up HA on my cluster. I consider my nodes as self-fencing, as they only have one single management NIC, one single CEPH NIC, one single power supply. Standard HA with NFS on a NAS works nicely, eg I pull the power cable from one of the nodes, the VMs in the HA config will be restarted on the other nodes.

Could CEPH recovery be tested in the same manner?

 

[aaron]

Sure, if it is not in production, and you are okay to run into problems, test all the failure scenarios you want to cover. If the guests are somewhat important, make backups first, in case you go too far and actually cause data loss.

But that is true for production environments as well. Always have backups. And do restore tests every once in a while

 

[robertut]

This is just a homelab set, running on three very nice HP Prodesk 600 G2 Mini (tiny form factor) boxes. The USB3 ports on it support 10G, so I was able to add 2.5G USB nics to each (and they do the transfer, with a bit of CPU tradeoff).

It runs a few home-related suff so yeah, kinda production but backps are made every night. The goal is to learn so yes, we need to run into problems to practice how to solve them.

BTW, discard and fstrim is indeed a key factor here, thanks for the tip. The CentOS VM I run doesn't seem to run fstrim on its own. After I ran it manually, and trimmed down a considerable amount of space, that immediately appeared as a decrease of occupied space on the CEPH storage graph. This has to be taken seriously. I've set up a cron job to do it every night, just before the VM backup.