[SOLVED] Adding a new separated pool to existing Ceph

M

matte.ragni

Active Member
Jan 15, 2021
2
0
41
36
I'm playing with a cluster composed by 3 machines with 3 hdd osd per machine (9 osd in total). It is a test environment to learn doing stuff, but I don't want to destroy it with this "expansion". I have a VM with an application that is not tolerating HDD slow performance, and since I have 3 ssd, I want to add another pool composed only by ssd and migrate the VM disk on this new pool (1 ssd per machine)

I found something relative the task on the doc (here and here on ceph doc). I think also this post is relevant, and I'm trying to derive the strategy to do it without destorying the existing pool if possible.

As for now I have 4 pools:
  • .mgr
  • ha-pool (the one I'm using for VM disk)
  • cephfs_data
  • cephfs_metadata
all with the CRUSH rule replicated_rule.

My strategy is the following:

  1. Force the existing replicated_ruleto accept only devices of hdd class. I have two way and I don't know which is better:
    1. creating a new rule (replicated_rule_hdd) with the command: ceph osd crush rule create-replicated replicated_rule_hdd default host hdd
    2. modifying the existing rule, but I have yet to understand if that is even possible... I think proxmox docs is telling me to do 1.
  2. If created new rule, edit in the interface the ha-pool Crush Rule (Advanced option in the "Edit: Ceph Pool" window) to select the new one. I'll do the same with also ceph_data.
  3. I'm strugling in understand what is the effect of point 2. Will it destroy something?
  4. I will start to add the ssd osd, one per machine, created with "Create: Ceph OSD" window, setting the (Advanced option) Device Class to ssd
  5. I will create a new rule replicated_rule_ssd with the command ceph osd crush rule create-replicated replicated_rule_ssd default host ssd
  6. Create a new pool ha-fast-pool added as storage with CRUSH rule replicated rule
  7. Optionally i could set the .mgr and cephfs-metadata for the CRUSH rule for ssd
  8. finally move the VM disk to the new ha-fast-pool storage

Do you see blocking?
Sorry for this bunch of confused ideas...
 
Overall it looks okay. But especially in the beginning it could be simplified.

create new HDD only rule:
Code: 
ceph osd crush rule create-replicated replicated_hdd default host hdd

Then assign that rule to the existing pools. Some rebalancing is possible. But your data is save.

Then add the SSDs and create the OSDs. Once you have OSDs with the new device class (nvme/ssd), you can create a rule specific for those device types:

Code: 
ceph osd crush rule create-replicated replicated_ssd default host ssd

If they are detected as nvme, adapt the rule accordingly. You can also freely specify the device class when you create a new OSD!

Then you can create the new pool and already assign the new rule right away.

Wheneve you use device class specific rules, it is important that every single pool is using a device class specific rule! Even the .mgr. Otherwise the autoscaler will not be able to calculate the optimal number of PGs for the pools, as there would be overlaps that cannot be resolved.

And when it comes to the optimal PG, do use the target_ratio settings to tell the autoscaler how much you expect that pool to consume in space, compared to the other pools (within the same device class). The .mgr doesn't need a target setting, it will stay at 1 PG.



https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pve_ceph_device_classes
 
You must log in or register to reply here.
Top Bottom
Back