Ceph usable space reporting

[kellogs]

I have a ceph cluster with 5 nodes. each node has 10x OSD and each OSD has 3.84TB space and replicate set to 3. Based on this I should have 64TB usable space


But when i looked at Ceph reporting from the proxmox gui it said this



why the difference?

 

[alexskysilk]

this is showing you raw capacity. you can use ceph df detail to get more granular data.

 

[kellogs]

Even the raw space is wrong isn’t it. It should be 192TB

 

[Neobin]

Even the raw space is wrong isn’t it. It should be 192TB

TB vs. TiB...

 

[LnxBil]

You should also keep in mind that you should never have more than 80% used in a ceph pool AND that you should leave space on each node so that a failed OSD can spill over to the other OSDs if you don't want to have a failed node.

To get more out of the system you currently have, consider using compression (e.g. lz4) to compress your data. This is negligible for performance, yet will increase the data stored on your system.

 

[kellogs]

yes I read about the 80% and I am planning to add 2 more ceph nodes with the exact same hardware spec. But I still don’t know why the reading is different