Ceph Health Warning

[powersupport]

Hi Team,

We are facing a Ceph health warning, please see the attached screenshot. Could you please advise us to fix this issue and please let us know why it was received?

Thank you

 

[mgabriel]

could you post the output of ceph status?

 

[powersupport]

ceph health shows warning

# ceph health
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time

 

[mgabriel]

Yes, but ceph status also shows how many other components there are. I guess you have a lot of OSDs, but I can only guess until you send the output of ceph status.

Ah, and if you like, you could also send the output of ceph health detail.

 

[powersupport]

Yes we have around 35 OSDs, below are the requested details

root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time

services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)

data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing

io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr

root@px-sata-sg1-n1:~#

 

[powersupport]

>ceph health detail

root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time

services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)

data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing

io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr

root@px-sata-sg1-n1:~# ceph health detail
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time
[WRN] PG_NOT_DEEP_SCRUBBED: 603 pgs not deep-scrubbed in time
pg 2.3fe not deep-scrubbed since 2024-02-26T18:25:10.007910+0800
pg 2.3fc not deep-scrubbed since 2024-03-02T05:04:17.867762+0800
pg 2.3fa not deep-scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not deep-scrubbed since 2024-02-26T18:55:25.608041+0800
pg 2.3f8 not deep-scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not deep-scrubbed since 2024-02-27T04:07:06.780463+0800
pg 2.3f6 not deep-scrubbed since 2024-03-01T20:09:32.579856+0800
pg 2.3f4 not deep-scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not deep-scrubbed since 2024-02-29T14:25:31.453597+0800
pg 2.3ee not deep-scrubbed since 2024-03-01T12:08:13.246636+0800
pg 2.3ed not deep-scrubbed since 2024-02-28T06:21:54.469762+0800
pg 2.3ec not deep-scrubbed since 2024-02-29T18:48:43.512844+0800
pg 2.3eb not deep-scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not deep-scrubbed since 2024-02-28T04:29:58.440340+0800
pg 2.3e3 not deep-scrubbed since 2024-03-02T10:55:30.667730+0800
pg 2.3e2 not deep-scrubbed since 2024-03-01T14:56:01.579726+0800
pg 2.3e0 not deep-scrubbed since 2024-03-03T03:53:48.552091+0800
pg 2.3de not deep-scrubbed since 2024-03-03T06:36:09.319970+0800
pg 2.3dc not deep-scrubbed since 2024-02-28T02:58:55.006370+0800
pg 2.3db not deep-scrubbed since 2024-03-01T20:45:56.557700+0800
pg 2.3da not deep-scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not deep-scrubbed since 2024-03-03T16:23:45.825107+0800
pg 2.3d8 not deep-scrubbed since 2024-03-03T01:58:13.748173+0800
pg 2.3d4 not deep-scrubbed since 2024-03-01T23:56:27.790808+0800
pg 2.3d2 not deep-scrubbed since 2024-03-03T04:31:05.436416+0800
pg 2.3d1 not deep-scrubbed since 2024-02-25T22:59:42.425663+0800
pg 2.3d0 not deep-scrubbed since 2024-02-29T00:26:17.722105+0800
pg 2.3cd not deep-scrubbed since 2024-02-26T00:39:14.417930+0800
pg 2.3cc not deep-scrubbed since 2024-03-02T01:12:48.109604+0800
pg 2.3cb not deep-scrubbed since 2024-03-01T07:35:25.396646+0800
pg 2.3ca not deep-scrubbed since 2024-03-03T03:17:09.849805+0800
pg 2.3c9 not deep-scrubbed since 2024-02-29T03:46:38.120981+0800
pg 2.3c8 not deep-scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not deep-scrubbed since 2024-02-28T12:02:39.273394+0800
pg 2.3c5 not deep-scrubbed since 2024-02-28T05:46:19.395063+0800
pg 2.3c4 not deep-scrubbed since 2024-03-03T03:50:38.278042+0800
pg 2.3c2 not deep-scrubbed since 2024-02-28T03:57:55.499343+0800
pg 2.3c1 not deep-scrubbed since 2024-02-27T07:21:23.521741+0800
pg 2.3bd not deep-scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not deep-scrubbed since 2024-03-01T02:33:03.717858+0800
pg 2.3ba not deep-scrubbed since 2024-02-28T22:57:54.364276+0800
pg 2.3b9 not deep-scrubbed since 2024-02-29T12:37:22.543559+0800
pg 2.3b8 not deep-scrubbed since 2024-03-03T12:27:43.247782+0800
pg 2.3b7 not deep-scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not deep-scrubbed since 2024-02-28T15:49:27.200217+0800
pg 2.3b5 not deep-scrubbed since 2024-02-27T18:48:57.542072+0800
pg 2.3af not deep-scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not deep-scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not deep-scrubbed since 2024-02-27T13:05:12.474032+0800
pg 2.3ab not deep-scrubbed since 2024-02-26T23:26:55.483502+0800
553 more pgs...
[WRN] PG_NOT_SCRUBBED: 603 pgs not scrubbed in time
pg 2.3fe not scrubbed since 2024-03-04T08:14:58.817228+0800
pg 2.3fc not scrubbed since 2024-03-05T05:11:51.871432+0800
pg 2.3fa not scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not scrubbed since 2024-03-04T10:14:31.379002+0800
pg 2.3f8 not scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not scrubbed since 2024-03-03T12:26:25.965874+0800
pg 2.3f6 not scrubbed since 2024-03-04T06:43:37.395523+0800
pg 2.3f4 not scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not scrubbed since 2024-03-04T13:20:30.502842+0800
pg 2.3ee not scrubbed since 2024-03-04T05:29:50.867175+0800
pg 2.3ed not scrubbed since 2024-03-04T22:35:40.804528+0800
pg 2.3ec not scrubbed since 2024-03-04T04:41:58.864418+0800
pg 2.3eb not scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not scrubbed since 2024-03-04T12:31:28.409116+0800
pg 2.3e3 not scrubbed since 2024-03-04T18:00:16.718806+0800
pg 2.3e2 not scrubbed since 2024-03-04T05:25:16.054478+0800
pg 2.3e0 not scrubbed since 2024-03-04T08:05:13.905426+0800
pg 2.3de not scrubbed since 2024-03-04T11:36:02.854784+0800
pg 2.3dc not scrubbed since 2024-03-04T08:31:47.511008+0800
pg 2.3db not scrubbed since 2024-03-05T00:41:56.080830+0800
pg 2.3da not scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not scrubbed since 2024-03-05T03:55:54.384114+0800
pg 2.3d8 not scrubbed since 2024-03-04T04:41:37.844753+0800
pg 2.3d4 not scrubbed since 2024-03-04T17:55:36.549616+0800
pg 2.3d2 not scrubbed since 2024-03-04T08:00:33.370522+0800
pg 2.3d1 not scrubbed since 2024-03-03T15:25:09.788452+0800
pg 2.3d0 not scrubbed since 2024-03-04T05:01:55.353989+0800
pg 2.3cd not scrubbed since 2024-03-03T16:51:49.013447+0800
pg 2.3cc not scrubbed since 2024-03-04T17:52:25.964289+0800
pg 2.3cb not scrubbed since 2024-03-03T16:56:12.798046+0800
pg 2.3ca not scrubbed since 2024-03-04T06:04:37.872510+0800
pg 2.3c9 not scrubbed since 2024-03-04T22:47:57.180631+0800
pg 2.3c8 not scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not scrubbed since 2024-03-04T19:37:51.044187+0800
pg 2.3c5 not scrubbed since 2024-03-04T11:43:06.678104+0800
pg 2.3c4 not scrubbed since 2024-03-04T20:15:50.165636+0800
pg 2.3c2 not scrubbed since 2024-03-04T19:22:16.316925+0800
pg 2.3c1 not scrubbed since 2024-03-05T00:03:16.565319+0800
pg 2.3bd not scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not scrubbed since 2024-03-04T23:19:13.508314+0800
pg 2.3ba not scrubbed since 2024-03-03T19:25:15.893331+0800
pg 2.3b9 not scrubbed since 2024-03-03T20:01:41.108637+0800
pg 2.3b8 not scrubbed since 2024-03-04T14:03:55.197692+0800
pg 2.3b7 not scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not scrubbed since 2024-03-04T22:36:43.567151+0800
pg 2.3b5 not scrubbed since 2024-03-05T01:10:26.388716+0800
pg 2.3af not scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not scrubbed since 2024-03-04T09:29:13.314556+0800
pg 2.3ab not scrubbed since 2024-03-03T07:44:35.921341+0800
553 more pgs...
root@px-sata-sg1-n1:~#

 

[powersupport]

Appreciate any help?


Thank you

 

[mgabriel]

168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep

Are these numbers always there and always like this?

do you use some kind of automated snapshot mechanism and slow, rotating disks in your Ceph?

 

[powersupport]

HI,

Currently the value standing on

633 pgs not deep-scrubbed in time

633 pgs not scrubbed in time

We don't use any mechanism to automate snapshot or anything

 

[mgabriel]

I've seen something similar recently. Could you please post the output of ceph osd pool autoscale-status?

 

[dtom]

are you by chance using HPE dl servers?

See here regarding snap trim issues