[SOLVED] Ceph misunderstanding

[gbarre]

Hi,

We have a cluster of 6 nodes, proxmox 6.1-8. Each node has 8 OSD of 750 GB.
After the installation (about a year ago), the available storage in the "Datacenter / summary" was about 12 TiB. Today it is 11.01 TiB.

When we look at the ceph storage part, we can see information that we do not understand : the total size is 8.94 TiB and it has decreased. (see the screenshot)

Could someone explain this to us please?

 

[Jakest]

You probably need to look at rebalancing your OSD's, as near as I can tell, ceph is trying to calculate space based on how much data you'll have when the first OSD fills up.

I've had this happen, and it was an OSD imbalance.

If there's a large imbalance, you may need to look at reweighting.

ceph osd reweight-by-utilization 120 (as an example)

https://docs.ceph.com/en/latest/rados/operations/control/

 

[gbarre]

That's what we did and it worked very well, we got our 12TB back.
Thank you very much.

 

[Jakest]

You might want to look into balancer as well, it's a more permanent solution