CEPH: uneven storage allocation on OSDs?

Dec 31, 2021
17
8
8
Hello together,
I'm just experimenting with CEPH and wondering why the OSDs are so unevenly allocated.

There are 7 PVE servers each with a 2TB and a 4TB NVMe.
I have ec 4+3 with the hosts configured as failure domain.

Does anyone have any idea if this is normal or if I should try to distribute this a bit better?

1686034949662.png
 
It seems like this is caused by the low amount of PGs you have per OSD. Did you set up the autoscaler for Ceph? You can do this by setting a target ratio on your Ceph pools. This can be done via our Web UI. More information about the value of the target ratio can be found in the Ceph docs [1]. If you have only one pool (on that device class), then setting it to 1 should be sufficient.

[1] https://docs.ceph.com/en/latest/rad...nt-groups/#viewing-pg-scaling-recommendations
 
  • Like
Reactions: qwertbert
Thanks, it looks like I need to enable the pg_autoscaler module first:
1686036893630.png

How can i do that? The pool was created with "PG Autoscale Mode on":

1686036840265.png
Edit: or is the module automatically activated when I set a target ratio for a pool?
Edit2: Modul is enabled:

Code:
root@pve1:~# ceph mgr module ls | grep pg_auto
pg_autoscaler      on (always on)
 
Last edited:
You need to check whether you have a target ratio (or size, but ratio is preferred) set for the pools. You can do this via

Code:
pveceph pool ls

(or via the Web UI)

If you have only one pool per storage class (seems like it, but I don't see the whole listing from your screenshot) then it is sufficient to set the target ratio for each pool to 1.



Otherwise, you need to set the target ratio relative to each other:

E.g. you have Pools A & B and expect Pool A to use 80% of space and Pool B to use 20% of space. Then you would set the target ratio for Pool A to 0.8 and for Pool B to 0.2
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!