ceph RBD PG # setting

[Ting]

Hi, all;

I have been running a 5 nodes ceph for a while now, I have 10 OSDs. I had one RBD drive setup with 128 pgs and auto scale on. My question is, since I have autoscale on, do I need to increase pg number on this RBD drive? or, proxmox ceph will automatically increase pg number for me?

 

[aaron]

I think there is some misunderstanding regarding the nomenclature

As far as I understand: You have a 2 node hyperconverged Proxmox VE + Ceph cluster with a total of 10 OSDs (2 per node?). You have one pool which is used for guest disk storage (RBD - rados block device).

The ideal number of PGs for that, in order to have around 100 PGs per OSD as seen in the output of ceph osd df tree, would be 256.

https://old.ceph.com/pgcalc/


You can set a target_size or target_ratio per pool. This will tell the autoscaler how much space overall the pool is expected to take up. It can then calculate the needed pg_num and set it for the expected size. If you don't do that, it can only use the current size of the pool, which might change.

If you do set the target_ratio of your pool to 1, you should see the autoscaler recommending 256 PGs. It will not change it automatically if the current pg_num is 128. This is because the autoscaler will only act by itself if the recommended pg_num differs by a factor of 3 from the current one. But you can still change it manually.

In the future, if the cluster changes, the autoscaler will then recalculate the best pg_num and either change it automatically or at least show you what value would be good. Things that have an influence are for example:

• adding more OSDs
• creating more Pools that also have a target_size or target_ratio