[SOLVED] Ceph: osd_max_backfills being overridden after changing it.

[rgj]

Hi,
I've added a new node to my cluster, but the backfilling to the new node is killing the cluster.
osd_max_backfills is set to 1000, so no wonder it kills it. I've tried setting osd_max_backfills on all nodes with:

ceph tell 'osd.*' injectargs --osd-max-backfills=1

But it doesn't change anything even if it seems to have succeeded.

I then tried changing it for one specific OSD.

# ceph daemon osd.20 config set osd_max_backfills 1
{
    "success": "osd_max_backfills = '1' "
}

But if I then fetch the value afterwards:

# ceph daemon osd.20 config get osd_max_backfills
{
    "osd_max_backfills": "1000"
}

I then tried setting it in ceph.conf and restarting the OSD. Right after the OSD came up the value was correct, but the next time is fetched the value, it was back at 1000.

root@phcpve03:~# ceph daemon osd.67 config get osd_max_backfills
{
    "osd_max_backfills": "1"
}
root@phcpve03:~# ceph daemon osd.67 config get osd_max_backfills
{
    "osd_max_backfills": "1000"
}

I vaguely remember messing with the parameter while setting up the cluster, but haven't been able to find anything in the config files.

Cheers,
René

 

[woifee]

Hi,
i'm struggling with exactly the same Problem. I'm on ceph Version 17.2.5 and i guess thats the key point. They changed the way these parameters work / are set. I'm still reading but i guess this documation will help us:
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/

Grüße
Wolfgang

 

[rgj]

Hi Wolfgang.

You are correct. Proxmox support sorted it out for me.

With 17.2 release the old parameters no longer works, as the default scheduler has been changed.

The weight parameters didn't do it for me, as it maxed out the 10Gbps link between the nodes.

Instead i used a combination of:
osd_mclock_scheduler_background_recovery_res=1
osd_mclock_scheduler_background_recovery_lim=100

The limit can be adjusted, but in our enviroment, that resulted in ~700MB/s recovery.

Cheers,
René

 

[woifee]

Hi René,

i can confirm that the weight parameters don't seem to have any effect.
I tried your parameters and values. They worked very well -> Thanks!

Cheers,
Wolfgang