Hardware replacement... slow OSD backfill...

[Rainerle]

Hi,
we are currently busy with replacing 8 servers after the leasing time is over with 8 new machines.

Had some problems with Broadcom 2x 100GBit BCM57508 P2100G BCM957508-P2100G and replaced them with the old but sturdy Mellanox cards so we are a little behind schedule with the OSD migration (Micron 9300 to KIOXIA).
Running Ceph Reef - what can you do to speed up the backfills???

 

[SteveITS]

Did you try turning off noout?

247 years seems like it’s not doing anything.

 

[Rainerle]

noout on or off does not seem to make a difference...
Current state:

 

[Rainerle]

Disabling noout and taking the old OSDs on one node down speed up the process.
We did set weight to 0 for the old OSDs, but that only took us so far...