CEPH: uneven storage allocation on OSDs?

[qwertbert]

Hello together,
I'm just experimenting with CEPH and wondering why the OSDs are so unevenly allocated.

There are 7 PVE servers each with a 2TB and a 4TB NVMe.
I have ec 4+3 with the hosts configured as failure domain.

Does anyone have any idea if this is normal or if I should try to distribute this a bit better?

 

[shanreich]

It seems like this is caused by the low amount of PGs you have per OSD. Did you set up the autoscaler for Ceph? You can do this by setting a target ratio on your Ceph pools. This can be done via our Web UI. More information about the value of the target ratio can be found in the Ceph docs [1]. If you have only one pool (on that device class), then setting it to 1 should be sufficient.

[1] https://docs.ceph.com/en/latest/rad...nt-groups/#viewing-pg-scaling-recommendations

 

[qwertbert]

Thanks, it looks like I need to enable the pg_autoscaler module first:


How can i do that? The pool was created with "PG Autoscale Mode on":


Edit: or is the module automatically activated when I set a target ratio for a pool?
Edit2: Modul is enabled:

root@pve1:~# ceph mgr module ls | grep pg_auto
pg_autoscaler      on (always on)

 

[shanreich]

You need to check whether you have a target ratio (or size, but ratio is preferred) set for the pools. You can do this via

pveceph pool ls

(or via the Web UI)

If you have only one pool per storage class (seems like it, but I don't see the whole listing from your screenshot) then it is sufficient to set the target ratio for each pool to 1.



Otherwise, you need to set the target ratio relative to each other:

E.g. you have Pools A & B and expect Pool A to use 80% of space and Pool B to use 20% of space. Then you would set the target ratio for Pool A to 0.8 and for Pool B to 0.2

 

[qwertbert]

Looks like I found the problem here (german).
The cluster is now rebalancing and the PG number is much higher.


Thanks for the push in the right direction.