backfillfull osd

[lcrmandapat]

Hi,

We are using Proxmox Virtual Environment 6.3-6. We are having problems with our ceph with the error below:

3 backfillfull osd(s)
osd.0 is backfill full
osd.1 is backfill full
osd.2 is backfill full

How can the "backfillfull" status be cleared?

Thank you in advance.

 

[VictorSTS]

Looks like some osd are too full. Post the full output of:

ceph df
ceph osd df
ceph status
ceph health detail
ceph osd dump | grep ratio

Please use code blocks when posting.

 

[lcrmandapat]

 

[VictorSTS]

Quite a late reply

Your backfill_full ratio is set to 0.91 and 2 of your 3 OSDs are filled to 92%: that's why ceph refuses to backfill to them.

osd.0 has a reweight set and that's why ceph won't place more data in it, even when it has nearly 394GiB free. reweight is 1 by default, so someone has changed that value at some point. You can try to set reweight to 1, as the other OSD, so some data will be moved from osd.1 and osd.2 to osd.0. Maybe it's safer or easier to use ceph osd reweight-by-utilization.

Regarding the recovery_unfound PG's maybe you have not lost data and just need to run a deep scrub of the affected PG's (looks like you are not running deep scrubs often).

In any case, get backups and test them before doing any operation. You don't want, ever, to get full disk in any case and even more with Ceph... and even more with such small setup with just 3 OSDs.

 

[lcrmandapat]

Hi @VictorSTS,

Thank you for your reply.

Yes, it's quite super late reply I thought its already fixed but now the problem recurs. We already performed deep scrubs the affected PGs.

We have backup of the VMs in our NAS in any case happens.

I appreciate your response.

We have another issue on deleting a VM here. Maybe you can also advice what we could do to remove the error and access the VM disks properly.

Thank you in advance.

 

[VictorSTS]

First thing would be to get your Ceph to a fully healthy state. Safety first, you know!

 

[lcrmandapat]

How can we make our Ceph to a fully healthy state? We already run the deep scrub but the error still persists.