Ceph OSD Balancing

[gdi2k]

I have a 3-node Ceph cluster configured as per the Proxmox wiki. Each node has 3 SSDs, as shown in the attached screenshot of the OSDs (2x 1TB and 1x 240 / 250GB).

I'm seeing quite a difference in usage between the drives; for example, osd.3 is 44.50% consumed, whereas osd.7 is just 32.07% consumed. I'm worried that, over time, this is going to cause problems - what happens when one drive reaches 100% and others are still at 75%?

I was hopeful that Ceph would be rebalancing these automatically, but apparently that's not how it works. Is there a standard way to rebalance manually without too much in the way of risk?

 

[udo]

Hi,
if you have not enough PGs (placement groups) in your pool, ceph distribute the data not equal over the OSDs.
But also with enough PGs the OSDs are not filled at the same level...

Udo

 

[gdi2k]

Thanks Udo.

I have 450 PGs for this pool, which I thought would be plenty based on the calculators.

Understand that data is not always distributed perfectly, but a 12% difference between the most and least used seems like a lot. Is that normal? It's manageable for now, but if it went up to 20 - 30%, it starts becoming an issue.

 

[udo]

Thanks Udo.

I have 450 PGs for this pool, which I thought would be plenty based on the calculators

Hi,
PGs should be ^2 - e.g. 512 was the right size for you.

Understand that data is not always distributed perfectly, but a 12% difference between the most and least used seems like a lot. Is that normal? It's manageable for now, but if it went up to 20 - 30%, it starts becoming an issue.

12% are not realy much - I have seen more…

Udo