Ceph pool config

maomaocake

Member
Feb 13, 2022
47
5
13
21
Hi
I currently have a pool of 3 1TB OSDs across 3 nodes Im planning to add 3 3tb hard disk and I was wondering if I should start a new pool with the 3tb hdds or add it to the existing pool

I only know that If I add to the pool depending on how I adjust the weights I either trade IO or capacity
 
Last edited:
As you already mentioned, adding OSDs with different capacity will cause more data to be stored on them, which will lead to performance degradation as they are hit more often by IO.

The other thing you should consider, especially in a 3-node cluster setup is the following.

Adding the new disks to the current cluster without making a new pool on them, you end up with 2 OSDs / node AFAIU.

What happens if not a full node, but only one OSD in one node fails?

Since you still have all 3 nodes in your cluster, Ceph will try to get back to 3 replicas (assuming size/min_size of 3/2). The remaining OSD will most likely get quite full, too full, unless your cluster is very empty. Running out of disk space is one of the few things you want to avoid at all costs with Ceph.

This is less of a problem with larger clusters, as Ceph can distribute that lost data among more nodes and still adhere to the "once replica per node" rule or if you have more OSDs per node in a 3-node cluster.

For this and the performance reasons, I suggest you create a new device class when you add those new OSDs. You can just enter a new name in the field in the GUI. Then create 2 rules, matching those 2 device classes and assign them to your pools.

The Ceph docs have a section on how to create such a rule: https://docs.ceph.com/en/latest/rados/operations/crush-map/#device-classes

For example: ceph osd crush rule create-replicated replicated_<my dev class> default host <my dev class>
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!