24 Scrub Errors, 3pgs inconsistent

[proxwolfe]

So I have been waiting for the last 1 pg to come out of deep scrubbing before I replace the next disk(s).

But it doesn't. Since I wrote this above, I have constantly had 1 pg being deep scrubbed (without knowing whether this is even true and, if it is, whether this is the same 1 pg all the time or whether it is a new one from time to time).

Is there a way to tell?

I would feel more comfortable replacing the next disk(s) from an "all green" position...

Thanks!

 

[sb-jw]

Unfortunately I don't have a cluster with a deep-scrub PG on hand at the moment, so I can't understand it in detail right now, but try ceph health detail or ceph pg dump pgs.

Since your cluster has been in an unhealthy state for a long time, the CEPH may now have to catch up on some scrubbing and it may indeed take some time.

 

[proxwolfe]

ceph health detail

Response: Health ok

ceph pg dump

Response: a super long list of all my PGs. I did find one that is - apparently - being deep scrubbed. It says the scrubbing has been going on for 143 seconds so far. I will check back a couple of times to see whether anything changes.

But there are also a few that have been scheduled for (deep) scrubbing. So it seems at least plausible that there is constantly (deep) scrubbing going on and likely it is not the same PG all the time.

Since your cluster has been in an unhealthy state for a long time, the CEPH may now have to catch up on some scrubbing and it may indeed take some time.

Exactly that was my thinking as well. It is just that it went down from 24 to 16 to 8 to 1 in a matter of hours/few days but it has been at 1 for almost two weeks now.