ceph pool can not used full

[Dzung]

I have proxmox with ceph, 42SSD 1.9TB, replicate 2, but my pool can not used full.
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77
TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB
on this show RAW is 77TB, if replicate 2 so pool used: 77TiB/2, but it show used 40TiB + MAX AVAIL 11TiB = 51TiB, so lost 26TiB
Can anyone please explain to me where the remaining storage goes

 

[ermanishchawla]

can you please share the output of the following

pveceph lspools to confirm you have pool with 3 copies or 2 copies

 

[Dzung]

can you please share the output of the following

pveceph lspools to confirm you have pool with 3 copies or 2 copies

thank you, this is my output,
pool 1 'pvedata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 18789 lfor 0/0/129 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps [1~4,6~b,12~12,25~1c,42~3,47~1,49~1a,64~2,67~1f,87~7,90~28,b9~2,bd~3,c5~4,ca~5,d0~8,d9~2,dc~1,de~5,e6~9,f0~1,f4~1,f6~3,fa~2,ff~2]

 

[ermanishchawla]

Replication factor of 2/1 is not recommended for production. Nevertheless can you share output of
ceph df
You already have some data in pool so remaining data only it will show as avail space

 

[Dzung]

we tested with 2/1, but I not understand why lost 26TiB

 

[ermanishchawla]

share the ceph df output

 

[Dzung]

this is my ceph df:
RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77 TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB

 

[ermanishchawla]

this is my ceph df:
RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77 TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB

According to this pool size is 77TB and if you use replication factor 2 with near full ratio of 85% available space would be 65.45 and if replication factor is 2, usa le space would be 32.75 TB approx and you have used 20TB of the available space, so according to this you must be left with nearly 12.75TB

This calculation is based on assumption near full ratio is normally configured as .85

Check your ceph osd tree and ceph osd dump to see more details

 

[Dzung]

According to this pool size is 77TB and if you use replication factor 2 with near full ratio of 85% available space would be 65.45 and if replication factor is 2, usa le space would be 32.75 TB approx and you have used 20TB of the available space, so according to this you must be left with nearly 12.75TB

This calculation is based on assumption near full ratio is normally configured as .85

Check your ceph osd tree and ceph osd dump to see more details

thank you!