ceph active+remapped+backfill_toofull recovery problem

[hbustos]

Hello Guys, I Have a recently installed Proxmox cluster whit 3 server, 10G NIC networking true a Aruba Swithc, 11 OSD SSD (8 x 1.7T, 3 x 3.4T), Total RAW capacity off 24T, 12T Used in 6 VM. One Pool

The problem are this 9 pg, that are active+remapped+backfill_toofull

Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull

pg 2.12 is active+remapped+backfill_toofull, acting [9,1,6] pg 2.2c is active+remapped+backfill_toofull, acting [9,7,2] pg 2.36 is active+remapped+backfill_toofull, acting [10,1,8] pg 2.39 is active+remapped+backfill_toofull, acting [9,6,2] pg 2.3f is active+remapped+backfill_toofull, acting [11,1,6] pg 2.5a is active+remapped+backfill_toofull, acting [11,6,2] pg 2.6e is active+remapped+backfill_toofull, acting [11,1,8] pg 2.71 is active+remapped+backfill_toofull, acting [11,8,2] pg 2.74 is active+remapped+backfill_toofull, acting [9,2,7]

This is the output for, ceph df
root@PVE12:~# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 24 TiB 12 TiB 12 TiB 12 TiB 50.03
TOTAL 24 TiB 12 TiB 12 TiB 12 TiB 50.03

--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 177 MiB 28 532 MiB 0.03 516 GiB
SSD1 2 128 4.3 TiB 1.19M 12 TiB 88.95 516 GiB
.rgw.root 3 32 1.3 KiB 4 48 KiB 0 516 GiB
default.rgw.log 4 32 242 B 2 32 KiB 0 516 GiB
default.rgw.control 5 32 0 B 8 0 B 0 516 GiB
default.rgw.meta 6 32 0 B 0 0 B 0 516 GiB

root@PVE12:~# ceph osd df tree


The recovery process stop, never reach more clean pg, the problem start with un spectect reboots of the physical servers, because and faulty UPS.

Really, apreciate any help. :

thanks

Heiber

 

[gurubert]

With replication size 3 you have a max usable capacity of 5.2 TB as this is the capacity of your smallest host.
Your OSDs are very unbalanced across the hosts which is not good for such a small cluster.
This is why the OSDs in pve10 reach the nearfull ratio and refuse to take any more data.

You could increase these ratios but you really need to balance the capacity across the nodes more evenly. Basically you are wasting 20TB in PVE12 because of this.

See https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

 

[hbustos]

Thanks, appreciate a lot, ¿You think I can change 3 copies for 2 and rebalance de pool ?

Heiber

 

[hbustos]

With replication size 3 you have a max usable capacity of 5.2 TB as this is the capacity of your smallest host.
Your OSDs are very unbalanced across the hosts which is not good for such a small cluster.
This is why the OSDs in pve10 reach the nearfull ratio and refuse to take any more data.

You could increase these ratios but you really need to balance the capacity across the nodes more evenly. Basically you are wasting 20TB in PVE12 because of this.

See https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

Thanks, appreciate a lot, ¿You think I can change 3 copies for 2 and rebalance de pool ?

Heiber

 

[gurubert]

You think I can change 3 copies for 2 and rebalance de pool ?

If you want to lose data, then yes.

 

[hbustos]

If you want to lose data, then yes.

Hello, thank for the advise.
we manage to resolve de warnings, the back_fill_toofull
by deleting information not used
and changing the set-backfillfull-ratio for all osd
finally the rebalance cut resolve the problems.

Now we have work to do, stop and destroy OSD, move to other servers for balance the osd in the servers. One by one.
Stell have 10% not rebalance, I think in the 91 PG Busy active+clean+remapped (see image)



thanks a lot, and any advise ?

Heiber

 

[gurubert]

Set the reweight value for the OSDs back to 1.0

 

[hbustos]

thanks for the advice, I change the weight for only one osd the osd.0, set to 1, and the pg busy increase, and the % rebuild decreases to 84% ¿This represent a problem?,

 

[gurubert]

This is because the data will be moved from other OSDs. Give Ceph time to do it.

 

[hbustos]

This is because the data will be moved from other OSDs. Give Ceph time to do it.

OK, thanks.
The Recovery /Rebalance process stop,
we have 313003/2575892 objects misplaced (12.151%),
this value increases wen I change the crush reweight to 1 for the osd.
¿That is normal?

 

[guruevi]

So from the output, you have 14TB in one server and 5TB in each of the other. With copies 3 and a failure domain set to host, you only have 5TB of capacity, for copies of 2 you still only have ~7.5T, not 12 What Ceph has been doing is placing data in unbalanced ways onto your “big” host, the problem that you have lost your capacity to rebalance without adding more hardware. Setting weights just tells Ceph which node to put more data on in comparison to others, typically Ceph figures that out on its own. Yes, objects are misplaced because they are in locations that if you lose the node you will lose data, but there is nowhere else to place the objects, setting the weights “properly” to 1 on each node will increase that number; it is telling you what to do, add more disks or nodes. You could potentially 1 by 1 remove a disk, and move it to another host, let Ceph rebuild until you have balanced disks but if this is professional work, it is cheaper and safer to just buy a set of SSD and plug them in.