Proxmox with Ceph: Ceph of Cluster show 19.97 TB but Ceph-vm shows only 4 TB.

[Bimlesh Singh]

I am running 2 proxmox clusters and in one cluster I did upgrade from 5.1 to 5.2. After upgrade I found that my ceph storage show wrong information. Ceph over all shows total storage of 19.97 TB where as Ceph-vm shows only 4.58 TB.
Before upgrade total storage and Ceph-vm storage show same vaule as 19.97 TB.

I am not sure where my other space disappeared. It is production servers so before I do any testing I need to know what exactly happen after upgrade. Please see the the Images to understand my problem more clearly.

 

[Alwin]

You can do a 'ceph df' and see what usage you have in your cluster. The change in the GUI reflects now this usage and not the global usage.

BTW: more then 3 MONs are only needed if you have thousands of ceph nodes/clients, anything else will just introduce latency and consume more resources.

 

[Bimlesh Singh]

Thank you for your reply. I want to get my full storage so that I can use my resources. what should I do to get it back as before.

 

[Alwin]

~20 TB / 3 = ~6.6 TB available space

This is the amount of data that you can put into your cluster. This data is replicated by the size of the pool (default size=3) and any other pool will take off of the available space too. Before it was showing the cluster total, this is not always correct and misleading, as you can not put that much data into your storage.

In your screenshot, you can see the global values as RAW capacity. While the pools below show the actual capacity for your data.

Thank you for your reply. I want to get my full storage so that I can use my resources. what should I do to get it back as before.

You can still see the total on the host -> ceph tab as seen in your previous picture.

 

[Bimlesh Singh]

~20 TB / 3 = ~6.6 TB available space

This is the amount of data that you can put into your cluster. This data is replicated by the size of the pool (default size=3) and any other pool will take off of the available space too. Before it was showing the cluster total, this is not always correct and misleading, as you can not put that much data into your storage.

In your screenshot, you can see the global values as RAW capacity. While the pools below show the actual capacity for your data.


You can still see the total on the host -> ceph tab as seen in your previous picture.

Thank you so much for your information. Can I reduce the default size = 3 to 2. If yes then is there any problem occur in future. As I understand this value 3 means 3 copies of 1 VM, how about if I want to keep 2 copies of 1 VM.

 

[Alwin]

Can I reduce the default size = 3 to 2.

No, you would need to create a new pool and move the data.

If yes then is there any problem occur in future.

On small setups the time needed to copy degraded objects is longer then on bigger setups, hence more risk of subsequent failures. This can lead to data loss.

As I understand this value 3 means 3 copies of 1 VM, how about if I want to keep 2 copies of 1 VM.

With two copies, the risk of losing data is greater, as you only have one PG left when a disk dies.

 

[Bimlesh Singh]

No, you would need to create a new pool and move the data.


On small setups the time needed to copy degraded objects is longer then on bigger setups, hence more risk of subsequent failures. This can lead to data loss.


With two copies, the risk of losing data is greater, as you only have one PG left when a disk dies.

Thanks for your clarification. I think it's time to close this case.