Proxmox/Ceph: Adding server with much larger drive pool.

[Waywatcher]

Good day all,

I have a 10 server Proxmox/Ceph cluster with 8 x 1.5T drives. I have 3 new servers with 24 x 3.5T drives. Is it safe/wise to add the new servers into the cluster? Or should they be placed in their own pool? If I add those drives I imagine I would have to modify the weight so that it uses more of the disk. I believe that otherwise it would use the same amount of space as the smaller drives. Also if the larger server(s) go(es) down and a re-balance happens that is ALOT more data than the smaller servers can probably balance without failing.

 

[aaron]

Ceph should automatically give the larger disks more weight. Your concern about one of those large nodes failing is definitely warranted for. I don't know your use case and how you want to use these nodes.

The options that come to my mind are:

• create a new independent 3 node cluster
• add them to the existing cluster, but create a new pool an make sure it is only using these nodes with a custom crush rule

For the latter, the easiest way is probably to define another custom device class and then create two new crush rules that specifically target "normal" smaller OSDs or the large disks. Then assign these rules to the pools accordingly.

Check out the Ceph docs regarding device classes and crush rules: https://docs.ceph.com/en/latest/rados/operations/crush-map/#device-classes

If you have a recent PVE installation (6.4) you can assign the created rules to the pools via the GUI.

 

[Waywatcher]

Thanks @aaron I will check out the docs.
I believe we will want to go with the same cluster different pool option.