scrubbing and deep-scrubbing, is this normal?

[nttec]

is it possible for proxmox to keep on scrubbing? I mean after one cycle of scrubbing then it start scrubbing again. from regular scrubbing then to deep-scrubbing.

we have osd.0 to osd.14

2019-11-25 15:45:34.453936 osd.13 osd.13 10.10.20.22:6800/2455 9484 : cluster [DBG] 1.143 deep-scrub ok
2019-11-25 15:45:35.931042 osd.1 osd.1 10.10.20.20:6804/2691 9373 : cluster [DBG] 1.141 deep-scrub starts

then after a while

2019-11-25 15:48:24.118305 osd.6 osd.6 10.10.20.21:6804/2769 5869 : cluster [DBG] 1.18c deep-scrub starts
2019-11-25 16:28:21.805064 osd.14 osd.14 10.10.20.22:6808/3081 9787 : cluster [DBG] 1.d0 scrub starts
2019-11-25 16:28:22.808230 osd.14 osd.14 10.10.20.22:6808/3081 9788 : cluster [DBG] 1.d0 scrub ok
2019-11-25 16:28:21.168032 osd.7 osd.7 10.10.20.21:6806/2889 7142 : cluster [DBG] 1.1fb deep-scrub ok
2019-11-25 16:28:24.005274 osd.1 osd.1 10.10.20.20:6804/2691 9377 : cluster [DBG] 1.de scrub starts
2019-11-25 16:28:24.889654 osd.1 osd.1 10.10.20.20:6804/2691 9378 : cluster [DBG] 1.de scrub ok

then after a while another one

2019-11-25 15:50:36.724863 osd.1 osd.1 10.10.20.20:6804/2691 9374 : cluster [DBG] 1.141 deep-scrub ok
2019-11-25 15:50:36.986733 osd.11 osd.11 10.10.20.22:6802/2583 8855 : cluster [DBG] 1.17e deep-scrub starts

and a lot of this in between

2019-11-25 16:07:24.145766 mgr.pxceph2 client.47824101 10.10.20.21:0/484391608 6553776 : cluster [DBG] pgmap v6554319: 512 pgs: 512 active+clean; 3.19TiB data, 9.37TiB used, 72.5TiB / 81.9TiB avail; 20.2MiB/s rd, 2.81MiB/s wr, 431op/s

this has been going on fordays now. Plus since we installed proxmox, we didn't setup a schedule for scrubbing and deep-scrubbing

UPDATE:

at one point I saw this

2019-11-25 16:00:00.000094 mon.pxceph mon.0 10.10.20.20:6789/0 6298908 : cluster [INF] overall HEALTH_OK

2019-11-25 17:05:45.126601 mon.pxceph mon.0 10.10.20.20:6789/0 6300749 : cluster [WRN] Health check failed: 3 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)

2019-11-25 17:05:50.526492 mon.pxceph mon.0 10.10.20.20:6789/0 6300753 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 3 slow requests are blocked > 32 sec. Implicated osds 3)

2019-11-25 17:05:50.526527 mon.pxceph mon.0 10.10.20.20:6789/0 6300754 : cluster [INF] Cluster is now healthy

but after that I still see lots of this

2019-11-25 16:09:51.597959 mgr.pxceph2 client.47824101 10.10.20.21:0/484391608 6553849 : cluster [DBG] pgmap v6554392: 512 pgs: 512 active+clean; 3.19TiB data, 9.37TiB used, 72.5TiB / 81.9TiB avail; 12.4MiB/s rd, 2.62MiB/s wr, 363op/s

and then shows a scrubbing start again.

2019-11-25 16:23:30.002722 osd.7 osd.7 10.10.20.21:6806/2889 7141 : cluster [DBG] 1.1fb deep-scrub starts

is this normal, should I be worry?

top is showing eash OSD daemon using 9 percent of memory and scrub/deep scrubbing is running everyday for some reason. Are these 2 related? Scrub settings has not been touch and left on default.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2441 ceph 20 0 6717436 5.712g 29272 S 4.3 9.1 4534:19 ceph-osd

 

[nttec]

anyone? any idea?

 

[nttec]

bumping on this

 

[tom]

See also https://docs.ceph.com/docs/master/architecture/

 

[nttec]

@tom - so this means I need to put up a scheduling for scrubbing, right?