scrubbing and deep-scrubbing, is this normal?

nttec

Well-Known Member
Jun 1, 2016
95
0
46
40
is it possible for proxmox to keep on scrubbing? I mean after one cycle of scrubbing then it start scrubbing again. from regular scrubbing then to deep-scrubbing.

we have osd.0 to osd.14


2019-11-25 15:45:34.453936 osd.13 osd.13 10.10.20.22:6800/2455 9484 : cluster [DBG] 1.143 deep-scrub ok
2019-11-25 15:45:35.931042 osd.1 osd.1 10.10.20.20:6804/2691 9373 : cluster [DBG] 1.141 deep-scrub starts

then after a while

2019-11-25 15:48:24.118305 osd.6 osd.6 10.10.20.21:6804/2769 5869 : cluster [DBG] 1.18c deep-scrub starts
2019-11-25 16:28:21.805064 osd.14 osd.14 10.10.20.22:6808/3081 9787 : cluster [DBG] 1.d0 scrub starts
2019-11-25 16:28:22.808230 osd.14 osd.14 10.10.20.22:6808/3081 9788 : cluster [DBG] 1.d0 scrub ok
2019-11-25 16:28:21.168032 osd.7 osd.7 10.10.20.21:6806/2889 7142 : cluster [DBG] 1.1fb deep-scrub ok
2019-11-25 16:28:24.005274 osd.1 osd.1 10.10.20.20:6804/2691 9377 : cluster [DBG] 1.de scrub starts
2019-11-25 16:28:24.889654 osd.1 osd.1 10.10.20.20:6804/2691 9378 : cluster [DBG] 1.de scrub ok


then after a while another one

2019-11-25 15:50:36.724863 osd.1 osd.1 10.10.20.20:6804/2691 9374 : cluster [DBG] 1.141 deep-scrub ok
2019-11-25 15:50:36.986733 osd.11 osd.11 10.10.20.22:6802/2583 8855 : cluster [DBG] 1.17e deep-scrub starts

and a lot of this in between

2019-11-25 16:07:24.145766 mgr.pxceph2 client.47824101 10.10.20.21:0/484391608 6553776 : cluster [DBG] pgmap v6554319: 512 pgs: 512 active+clean; 3.19TiB data, 9.37TiB used, 72.5TiB / 81.9TiB avail; 20.2MiB/s rd, 2.81MiB/s wr, 431op/s

this has been going on fordays now. Plus since we installed proxmox, we didn't setup a schedule for scrubbing and deep-scrubbing

UPDATE:

at one point I saw this

2019-11-25 16:00:00.000094 mon.pxceph mon.0 10.10.20.20:6789/0 6298908 : cluster [INF] overall HEALTH_OK

2019-11-25 17:05:45.126601 mon.pxceph mon.0 10.10.20.20:6789/0 6300749 : cluster [WRN] Health check failed: 3 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)

2019-11-25 17:05:50.526492 mon.pxceph mon.0 10.10.20.20:6789/0 6300753 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 3 slow requests are blocked > 32 sec. Implicated osds 3)

2019-11-25 17:05:50.526527 mon.pxceph mon.0 10.10.20.20:6789/0 6300754 : cluster [INF] Cluster is now healthy

but after that I still see lots of this

2019-11-25 16:09:51.597959 mgr.pxceph2 client.47824101 10.10.20.21:0/484391608 6553849 : cluster [DBG] pgmap v6554392: 512 pgs: 512 active+clean; 3.19TiB data, 9.37TiB used, 72.5TiB / 81.9TiB avail; 12.4MiB/s rd, 2.62MiB/s wr, 363op/s


and then shows a scrubbing start again.

2019-11-25 16:23:30.002722 osd.7 osd.7 10.10.20.21:6806/2889 7141 : cluster [DBG] 1.1fb deep-scrub starts

is this normal, should I be worry?

top is showing eash OSD daemon using 9 percent of memory and scrub/deep scrubbing is running everyday for some reason. Are these 2 related? Scrub settings has not been touch and left on default.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2441 ceph 20 0 6717436 5.712g 29272 S 4.3 9.1 4534:19 ceph-osd
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!