ceph pool can not used full

Dzung

Well-Known Member
Feb 20, 2019
84
2
48
33
I have proxmox with ceph, 42SSD 1.9TB, replicate 2, but my pool can not used full.
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77
TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB
on this show RAW is 77TB, if replicate 2 so pool used: 77TiB/2, but it show used 40TiB + MAX AVAIL 11TiB = 51TiB, so lost 26TiB
Can anyone please explain to me where the remaining storage goes
 
can you please share the output of the following

pveceph lspools to confirm you have pool with 3 copies or 2 copies
 
can you please share the output of the following

pveceph lspools to confirm you have pool with 3 copies or 2 copies
thank you, this is my output,
pool 1 'pvedata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 18789 lfor 0/0/129 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps [1~4,6~b,12~12,25~1c,42~3,47~1,49~1a,64~2,67~1f,87~7,90~28,b9~2,bd~3,c5~4,ca~5,d0~8,d9~2,dc~1,de~5,e6~9,f0~1,f4~1,f6~3,fa~2,ff~2]
 
Replication factor of 2/1 is not recommended for production. Nevertheless can you share output of
ceph df
You already have some data in pool so remaining data only it will show as avail space
 
this is my ceph df:
RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77 TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB
 
this is my ceph df:
RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 77 TiB 34 TiB 43 TiB 43 TiB 56.11 77 TOTAL 77 TiB 34 TiB 43 TiB 43 TiB 56.11 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL pvedata 1 20 TiB 5.94M 40 TiB 63.51 11 TiB
According to this pool size is 77TB and if you use replication factor 2 with near full ratio of 85% available space would be 65.45 and if replication factor is 2, usa le space would be 32.75 TB approx and you have used 20TB of the available space, so according to this you must be left with nearly 12.75TB

This calculation is based on assumption near full ratio is normally configured as .85

Check your ceph osd tree and ceph osd dump to see more details
 
  • Like
Reactions: Dzung
According to this pool size is 77TB and if you use replication factor 2 with near full ratio of 85% available space would be 65.45 and if replication factor is 2, usa le space would be 32.75 TB approx and you have used 20TB of the available space, so according to this you must be left with nearly 12.75TB

This calculation is based on assumption near full ratio is normally configured as .85

Check your ceph osd tree and ceph osd dump to see more details
thank you!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!