Strange behavior of the ceph space

[Sean510]

Hello,

i am left with this problem and cannot find a solution. I don't know what to watch anymore.
I did not find orphaned items. Can you advise me where to look?

Ceph version 16.2.6

Best regards


root@#####:~# rbd -p cl1-storage du NAME PROVISIONED USED vm-100-disk-0 1 TiB 213 GiB vm-103-disk-0 2 TiB 1.7 TiB vm-110-disk-0 250 GiB 26 GiB vm-110-disk-1 125 GiB 75 GiB vm-110-disk-2 125 GiB 9.0 GiB vm-112-disk-0 20 GiB 20 GiB vm-113-disk-0 2 TiB 1.6 TiB vm-117-disk-0 2.0 TiB 2.0 TiB vm-134-disk-0 1 TiB 346 GiB vm-138-disk-0 50 GiB 39 GiB vm-142-disk-0 500 GiB 127 GiB vm-142-disk-1 1 MiB 1 MiB vm-145-disk-0 1 TiB 993 GiB vm-156-disk-0 50 GiB 48 GiB vm-156-disk-1 1 MiB 1 MiB vm-178-disk-0 1 TiB 361 GiB vm-187-disk-1 1 MiB 1 MiB <TOTAL> 11 TiB 7.4 TiB

root@######:~# ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 105 TiB 42 TiB 63 TiB 63 TiB 60.32 TOTAL 105 TiB 42 TiB 63 TiB 63 TiB 60.32 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 2 0 B 0 B 0 B 0 0 B 0 B 0 B 0 11 TiB N/A N/A N/A 0 B 0 B cl1-storage 2 512 86 TiB 86 TiB 12 MiB 25.22M 62 TiB 62 TiB 37 MiB 66.36 11 TiB N/A N/A N/A 0 B 0 B .rgw.root 3 32 1.3 KiB 1.3 KiB 0 B 4 48 KiB 48 KiB 0 B 0 11 TiB N/A N/A N/A 0 B 0 B default.rgw.log 4 32 3.6 KiB 3.6 KiB 0 B 34 408 KiB 408 KiB 0 B 0 11 TiB N/A N/A N/A 0 B 0 B default.rgw.control 5 32 0 B 0 B 0 B 8 0 B 0 B 0 B 0 11 TiB N/A N/A N/A 0 B 0 B default.rgw.meta 6 8 0 B 0 B 0 B 0 0 B 0 B 0 B 0 11 TiB N/A N/A N/A 0 B 0 B

 

[datschlatscher]

Could you elaborate what your problem exactly is?
From your screenshots and log outputs there doesn't seem to be anything out of the ordinary. Are you experiencing high disk usage?

How are you going about finding and deleting orphaned files? If you look at the documentation ORPHAN LIST AND ASSOCIATED TOOLING it says that a file should be created containing information about orphaned files (this however might take a while).

 

[Sean510]

After investigating thoroughly, we were able to find that ceph left a lot of raw.data not deleted. All these data refer to processes of copy, move, clone, etc. etc. As there was no manual deletion solution, I emptied the pool and recreated from scratch.

pool image num objects prefix
cl1-storage vm-100-disk-0 262144 rbd_data.5ed48bc35c985f
cl1-storage vm-103-disk-0 524288 rbd_data.1e84e6920143c
cl1-storage vm-112-disk-0 5120 rbd_data.2153741d67125a
cl1-storage vm-113-disk-0 524288 rbd_data.4aa5dca4c4550
cl1-storage vm-117-disk-0 518144 rbd_data.1703574e7eb5
cl1-storage vm-142-disk-0 128000 rbd_data.32efb5cad0889e
cl1-storage vm-142-disk-1 1 rbd_data.32de46f322336e
cl1-storage vm-145-disk-0 262144 rbd_data.6eed8c23cd8e2
cl1-storage vm-156-disk-0 12800 rbd_data.320068a158691
cl1-storage vm-156-disk-1 1 rbd_data.32af667d2eba86
cl1-storage vm-178-disk-0 262144 rbd_data.1488bd9205972
cl1-storage vm-187-disk-1 1 rbd_data.28425729324fb9


cl1-storage 47 NOT rbd_data
tot 2499122
cl1-storage all 25162000
diff 22662878

 

[Sean510]

even after recreating everything, the problem remains
clones and copies are not deleted

rados -p storage df
POOL_NAME      USED  OBJECTS  CLONES   COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED     RD_OPS       RD     WR_OPS      WR  USED COMPR  UNDER COMPR
storage  25 TiB  2533374  181180  7600122                   0        0         0  249391505  1.9 TiB  122760588  15 TiB         0 B          0 B

total_objects    2533374
total_used       25 TiB
total_avail      80 TiB
total_space      105 TiB