How are people handling ceph v19 osd fragmentation?

S

stemid

New Member
Proxmox Subscriber
Apr 30, 2025
13
0
1
When we first upgraded to Proxmox v9 and Ceph v19 we changed the fragmentation threshold from 8 to 9 to mute a bunch of health warnings.

But now an OSD is actually over 0.9, which according to some sources could be considered serious. And of course it sets our entire cluster into HEALTH_WARN, which triggers alerts.

Code: 
HEALTH_WARN 1 OSD(s)
[WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s)
     osd.9 0.903071

I tried doing an evacuate/refill using reweight on osd.9, waiting until everything was recovered, then reweight again to refill osd.9, but the fragmentation value only went down a bit, it's still over 0.9.

I don't want to hear the solution is "ceph health mute BLUESTORE_FREE_FRAGMENTATION 168h", that's terribly unprofessional, there has to be a proper solution to this.

The last resort seems to be to completely delete the osd, and then re-create it. Hoping to solve fragmentation by re-writing all the data on the osd.

What have people on the forum done with this issue? I tried searching for BLUESTORE_FREE_FRAGMENTATION but only found one relevant thread and it was in german.
 
Last edited:
Ok this sounds like solid advice, unfortunately it's difficult to execute when we only ever had one pool and it's only used for proxmox VMs. We never planned for which VMs would have more objects, and where to place them.

Is that a best practice? To create multiple pools and place VMs based on what number of file objects they might have?

Right now we have three OSDs warning about fragmentation, it's gone up since I posted this.

But this cluster has been working fine for years before v19 came along.

I hate to do this but at this time our only course of action is to mute the BLUESTORE_FREE_FRAGMENTATION until we have time to plan for a new pool, and migrating some busy VMs to that pool.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back