ceph backfill problem

[Allen Chen]

Hi there,

Has anyone experienced below?
2 of OSD server was down, after bring up 2 of servers, I brought 52 OSD's in with just weight of 0.05, but it causing huge backfilling load, I saw so many blocked requests and a number of pg stuck inactive. some of servers was impact. so I stopped backfilling by mark nobackfill flag. everything back to normal.
But the most strange thing happens after 2 hours, the backfilling suddenly start again despite of nobackfill flag marked and causing so many blocked requests then we have to reweight 52 OSD's to 0 to stabilize storage.

Not sure why backfill start again. Anyone has any idea about that please comments.

Thanks so much.
Allen

 

[sg90]

Did you have any OSD's that went down / out or anything that happened to the rest of the cluster?

 

[Allen Chen]

Did you have any OSD's that went down / out or anything that happened to the rest of the cluster?

nope, no other OSD down/out at the moment.

 

[spirit]

do you have the default value "osd max backfills = 1" ?

 

[Alwin]

nope, no other OSD down/out at the moment.

check your ceph logs, it might happen very quick.

 

[Allen Chen]

do you have the default value "osd max backfills = 1" ?

Thanks.
Yes, I set max backfills as 1 to minimize backfill, then second backfill happen despite of nobackfill flagged.

 

[Allen Chen]

Anyone has any idea or encounter same issue? Please comments, Thanks a lot.