Ceph Health Warning
- Thread starter powersupport
- Start date
ceph health shows warning
# ceph health
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time
# ceph health
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time
Yes, but
Ah, and if you like, you could also send the output of
ceph status also shows how many other components there are. I guess you have a lot of OSDs, but I can only guess until you send the output of ceph status.Ah, and if you like, you could also send the output of
ceph health detail.
Yes we have around 35 OSDs, below are the requested details
root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time
services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)
data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing
io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr
root@px-sata-sg1-n1:~#
root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time
services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)
data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing
io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr
root@px-sata-sg1-n1:~#
Last edited:
>ceph health detail
root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time
services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)
data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing
io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr
root@px-sata-sg1-n1:~# ceph health detail
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time
[WRN] PG_NOT_DEEP_SCRUBBED: 603 pgs not deep-scrubbed in time
pg 2.3fe not deep-scrubbed since 2024-02-26T18:25:10.007910+0800
pg 2.3fc not deep-scrubbed since 2024-03-02T05:04:17.867762+0800
pg 2.3fa not deep-scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not deep-scrubbed since 2024-02-26T18:55:25.608041+0800
pg 2.3f8 not deep-scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not deep-scrubbed since 2024-02-27T04:07:06.780463+0800
pg 2.3f6 not deep-scrubbed since 2024-03-01T20:09:32.579856+0800
pg 2.3f4 not deep-scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not deep-scrubbed since 2024-02-29T14:25:31.453597+0800
pg 2.3ee not deep-scrubbed since 2024-03-01T12:08:13.246636+0800
pg 2.3ed not deep-scrubbed since 2024-02-28T06:21:54.469762+0800
pg 2.3ec not deep-scrubbed since 2024-02-29T18:48:43.512844+0800
pg 2.3eb not deep-scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not deep-scrubbed since 2024-02-28T04:29:58.440340+0800
pg 2.3e3 not deep-scrubbed since 2024-03-02T10:55:30.667730+0800
pg 2.3e2 not deep-scrubbed since 2024-03-01T14:56:01.579726+0800
pg 2.3e0 not deep-scrubbed since 2024-03-03T03:53:48.552091+0800
pg 2.3de not deep-scrubbed since 2024-03-03T06:36:09.319970+0800
pg 2.3dc not deep-scrubbed since 2024-02-28T02:58:55.006370+0800
pg 2.3db not deep-scrubbed since 2024-03-01T20:45:56.557700+0800
pg 2.3da not deep-scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not deep-scrubbed since 2024-03-03T16:23:45.825107+0800
pg 2.3d8 not deep-scrubbed since 2024-03-03T01:58:13.748173+0800
pg 2.3d4 not deep-scrubbed since 2024-03-01T23:56:27.790808+0800
pg 2.3d2 not deep-scrubbed since 2024-03-03T04:31:05.436416+0800
pg 2.3d1 not deep-scrubbed since 2024-02-25T22:59:42.425663+0800
pg 2.3d0 not deep-scrubbed since 2024-02-29T00:26:17.722105+0800
pg 2.3cd not deep-scrubbed since 2024-02-26T00:39:14.417930+0800
pg 2.3cc not deep-scrubbed since 2024-03-02T01:12:48.109604+0800
pg 2.3cb not deep-scrubbed since 2024-03-01T07:35:25.396646+0800
pg 2.3ca not deep-scrubbed since 2024-03-03T03:17:09.849805+0800
pg 2.3c9 not deep-scrubbed since 2024-02-29T03:46:38.120981+0800
pg 2.3c8 not deep-scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not deep-scrubbed since 2024-02-28T12:02:39.273394+0800
pg 2.3c5 not deep-scrubbed since 2024-02-28T05:46:19.395063+0800
pg 2.3c4 not deep-scrubbed since 2024-03-03T03:50:38.278042+0800
pg 2.3c2 not deep-scrubbed since 2024-02-28T03:57:55.499343+0800
pg 2.3c1 not deep-scrubbed since 2024-02-27T07:21:23.521741+0800
pg 2.3bd not deep-scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not deep-scrubbed since 2024-03-01T02:33:03.717858+0800
pg 2.3ba not deep-scrubbed since 2024-02-28T22:57:54.364276+0800
pg 2.3b9 not deep-scrubbed since 2024-02-29T12:37:22.543559+0800
pg 2.3b8 not deep-scrubbed since 2024-03-03T12:27:43.247782+0800
pg 2.3b7 not deep-scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not deep-scrubbed since 2024-02-28T15:49:27.200217+0800
pg 2.3b5 not deep-scrubbed since 2024-02-27T18:48:57.542072+0800
pg 2.3af not deep-scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not deep-scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not deep-scrubbed since 2024-02-27T13:05:12.474032+0800
pg 2.3ab not deep-scrubbed since 2024-02-26T23:26:55.483502+0800
553 more pgs...
[WRN] PG_NOT_SCRUBBED: 603 pgs not scrubbed in time
pg 2.3fe not scrubbed since 2024-03-04T08:14:58.817228+0800
pg 2.3fc not scrubbed since 2024-03-05T05:11:51.871432+0800
pg 2.3fa not scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not scrubbed since 2024-03-04T10:14:31.379002+0800
pg 2.3f8 not scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not scrubbed since 2024-03-03T12:26:25.965874+0800
pg 2.3f6 not scrubbed since 2024-03-04T06:43:37.395523+0800
pg 2.3f4 not scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not scrubbed since 2024-03-04T13:20:30.502842+0800
pg 2.3ee not scrubbed since 2024-03-04T05:29:50.867175+0800
pg 2.3ed not scrubbed since 2024-03-04T22:35:40.804528+0800
pg 2.3ec not scrubbed since 2024-03-04T04:41:58.864418+0800
pg 2.3eb not scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not scrubbed since 2024-03-04T12:31:28.409116+0800
pg 2.3e3 not scrubbed since 2024-03-04T18:00:16.718806+0800
pg 2.3e2 not scrubbed since 2024-03-04T05:25:16.054478+0800
pg 2.3e0 not scrubbed since 2024-03-04T08:05:13.905426+0800
pg 2.3de not scrubbed since 2024-03-04T11:36:02.854784+0800
pg 2.3dc not scrubbed since 2024-03-04T08:31:47.511008+0800
pg 2.3db not scrubbed since 2024-03-05T00:41:56.080830+0800
pg 2.3da not scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not scrubbed since 2024-03-05T03:55:54.384114+0800
pg 2.3d8 not scrubbed since 2024-03-04T04:41:37.844753+0800
pg 2.3d4 not scrubbed since 2024-03-04T17:55:36.549616+0800
pg 2.3d2 not scrubbed since 2024-03-04T08:00:33.370522+0800
pg 2.3d1 not scrubbed since 2024-03-03T15:25:09.788452+0800
pg 2.3d0 not scrubbed since 2024-03-04T05:01:55.353989+0800
pg 2.3cd not scrubbed since 2024-03-03T16:51:49.013447+0800
pg 2.3cc not scrubbed since 2024-03-04T17:52:25.964289+0800
pg 2.3cb not scrubbed since 2024-03-03T16:56:12.798046+0800
pg 2.3ca not scrubbed since 2024-03-04T06:04:37.872510+0800
pg 2.3c9 not scrubbed since 2024-03-04T22:47:57.180631+0800
pg 2.3c8 not scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not scrubbed since 2024-03-04T19:37:51.044187+0800
pg 2.3c5 not scrubbed since 2024-03-04T11:43:06.678104+0800
pg 2.3c4 not scrubbed since 2024-03-04T20:15:50.165636+0800
pg 2.3c2 not scrubbed since 2024-03-04T19:22:16.316925+0800
pg 2.3c1 not scrubbed since 2024-03-05T00:03:16.565319+0800
pg 2.3bd not scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not scrubbed since 2024-03-04T23:19:13.508314+0800
pg 2.3ba not scrubbed since 2024-03-03T19:25:15.893331+0800
pg 2.3b9 not scrubbed since 2024-03-03T20:01:41.108637+0800
pg 2.3b8 not scrubbed since 2024-03-04T14:03:55.197692+0800
pg 2.3b7 not scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not scrubbed since 2024-03-04T22:36:43.567151+0800
pg 2.3b5 not scrubbed since 2024-03-05T01:10:26.388716+0800
pg 2.3af not scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not scrubbed since 2024-03-04T09:29:13.314556+0800
pg 2.3ab not scrubbed since 2024-03-03T07:44:35.921341+0800
553 more pgs...
root@px-sata-sg1-n1:~#
root@px-sata-sg1-n1:~# ceph status
cluster:
id: 8b561bdc-3821-4059-a15d-4a63b3bce13c
health: HEALTH_WARN
603 pgs not deep-scrubbed in time
603 pgs not scrubbed in time
services:
mon: 3 daemons, quorum px-sata-sg1-n2,px-sata-sg1-n3,px-sata-sg1-n4 (age 9w)
mgr: px-sata-sg1-n2(active, since 3M), standbys: px-sata-sg1-n3
osd: 35 osds: 35 up (since 11w), 35 in (since 3M)
data:
pools: 2 pools, 1025 pgs
objects: 11.08M objects, 35 TiB
usage: 99 TiB used, 64 TiB / 163 TiB avail
pgs: 764 active+clean
168 active+clean+snaptrim_wait
83 active+clean+scrubbing+deep
9 active+clean+snaptrim
1 active+clean+scrubbing
io:
client: 61 MiB/s rd, 895 KiB/s wr, 545 op/s rd, 81 op/s wr
root@px-sata-sg1-n1:~# ceph health detail
HEALTH_WARN 603 pgs not deep-scrubbed in time; 603 pgs not scrubbed in time
[WRN] PG_NOT_DEEP_SCRUBBED: 603 pgs not deep-scrubbed in time
pg 2.3fe not deep-scrubbed since 2024-02-26T18:25:10.007910+0800
pg 2.3fc not deep-scrubbed since 2024-03-02T05:04:17.867762+0800
pg 2.3fa not deep-scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not deep-scrubbed since 2024-02-26T18:55:25.608041+0800
pg 2.3f8 not deep-scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not deep-scrubbed since 2024-02-27T04:07:06.780463+0800
pg 2.3f6 not deep-scrubbed since 2024-03-01T20:09:32.579856+0800
pg 2.3f4 not deep-scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not deep-scrubbed since 2024-02-29T14:25:31.453597+0800
pg 2.3ee not deep-scrubbed since 2024-03-01T12:08:13.246636+0800
pg 2.3ed not deep-scrubbed since 2024-02-28T06:21:54.469762+0800
pg 2.3ec not deep-scrubbed since 2024-02-29T18:48:43.512844+0800
pg 2.3eb not deep-scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not deep-scrubbed since 2024-02-28T04:29:58.440340+0800
pg 2.3e3 not deep-scrubbed since 2024-03-02T10:55:30.667730+0800
pg 2.3e2 not deep-scrubbed since 2024-03-01T14:56:01.579726+0800
pg 2.3e0 not deep-scrubbed since 2024-03-03T03:53:48.552091+0800
pg 2.3de not deep-scrubbed since 2024-03-03T06:36:09.319970+0800
pg 2.3dc not deep-scrubbed since 2024-02-28T02:58:55.006370+0800
pg 2.3db not deep-scrubbed since 2024-03-01T20:45:56.557700+0800
pg 2.3da not deep-scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not deep-scrubbed since 2024-03-03T16:23:45.825107+0800
pg 2.3d8 not deep-scrubbed since 2024-03-03T01:58:13.748173+0800
pg 2.3d4 not deep-scrubbed since 2024-03-01T23:56:27.790808+0800
pg 2.3d2 not deep-scrubbed since 2024-03-03T04:31:05.436416+0800
pg 2.3d1 not deep-scrubbed since 2024-02-25T22:59:42.425663+0800
pg 2.3d0 not deep-scrubbed since 2024-02-29T00:26:17.722105+0800
pg 2.3cd not deep-scrubbed since 2024-02-26T00:39:14.417930+0800
pg 2.3cc not deep-scrubbed since 2024-03-02T01:12:48.109604+0800
pg 2.3cb not deep-scrubbed since 2024-03-01T07:35:25.396646+0800
pg 2.3ca not deep-scrubbed since 2024-03-03T03:17:09.849805+0800
pg 2.3c9 not deep-scrubbed since 2024-02-29T03:46:38.120981+0800
pg 2.3c8 not deep-scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not deep-scrubbed since 2024-02-28T12:02:39.273394+0800
pg 2.3c5 not deep-scrubbed since 2024-02-28T05:46:19.395063+0800
pg 2.3c4 not deep-scrubbed since 2024-03-03T03:50:38.278042+0800
pg 2.3c2 not deep-scrubbed since 2024-02-28T03:57:55.499343+0800
pg 2.3c1 not deep-scrubbed since 2024-02-27T07:21:23.521741+0800
pg 2.3bd not deep-scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not deep-scrubbed since 2024-03-01T02:33:03.717858+0800
pg 2.3ba not deep-scrubbed since 2024-02-28T22:57:54.364276+0800
pg 2.3b9 not deep-scrubbed since 2024-02-29T12:37:22.543559+0800
pg 2.3b8 not deep-scrubbed since 2024-03-03T12:27:43.247782+0800
pg 2.3b7 not deep-scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not deep-scrubbed since 2024-02-28T15:49:27.200217+0800
pg 2.3b5 not deep-scrubbed since 2024-02-27T18:48:57.542072+0800
pg 2.3af not deep-scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not deep-scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not deep-scrubbed since 2024-02-27T13:05:12.474032+0800
pg 2.3ab not deep-scrubbed since 2024-02-26T23:26:55.483502+0800
553 more pgs...
[WRN] PG_NOT_SCRUBBED: 603 pgs not scrubbed in time
pg 2.3fe not scrubbed since 2024-03-04T08:14:58.817228+0800
pg 2.3fc not scrubbed since 2024-03-05T05:11:51.871432+0800
pg 2.3fa not scrubbed since 2024-03-04T09:13:27.759277+0800
pg 2.3f9 not scrubbed since 2024-03-04T10:14:31.379002+0800
pg 2.3f8 not scrubbed since 2024-03-04T03:02:32.468241+0800
pg 2.3f7 not scrubbed since 2024-03-03T12:26:25.965874+0800
pg 2.3f6 not scrubbed since 2024-03-04T06:43:37.395523+0800
pg 2.3f4 not scrubbed since 2024-03-05T05:13:46.677910+0800
pg 2.3f3 not scrubbed since 2024-03-04T13:20:30.502842+0800
pg 2.3ee not scrubbed since 2024-03-04T05:29:50.867175+0800
pg 2.3ed not scrubbed since 2024-03-04T22:35:40.804528+0800
pg 2.3ec not scrubbed since 2024-03-04T04:41:58.864418+0800
pg 2.3eb not scrubbed since 2024-03-04T17:52:10.554873+0800
pg 2.3e8 not scrubbed since 2024-03-04T12:31:28.409116+0800
pg 2.3e3 not scrubbed since 2024-03-04T18:00:16.718806+0800
pg 2.3e2 not scrubbed since 2024-03-04T05:25:16.054478+0800
pg 2.3e0 not scrubbed since 2024-03-04T08:05:13.905426+0800
pg 2.3de not scrubbed since 2024-03-04T11:36:02.854784+0800
pg 2.3dc not scrubbed since 2024-03-04T08:31:47.511008+0800
pg 2.3db not scrubbed since 2024-03-05T00:41:56.080830+0800
pg 2.3da not scrubbed since 2024-03-03T21:22:43.007960+0800
pg 2.3d9 not scrubbed since 2024-03-05T03:55:54.384114+0800
pg 2.3d8 not scrubbed since 2024-03-04T04:41:37.844753+0800
pg 2.3d4 not scrubbed since 2024-03-04T17:55:36.549616+0800
pg 2.3d2 not scrubbed since 2024-03-04T08:00:33.370522+0800
pg 2.3d1 not scrubbed since 2024-03-03T15:25:09.788452+0800
pg 2.3d0 not scrubbed since 2024-03-04T05:01:55.353989+0800
pg 2.3cd not scrubbed since 2024-03-03T16:51:49.013447+0800
pg 2.3cc not scrubbed since 2024-03-04T17:52:25.964289+0800
pg 2.3cb not scrubbed since 2024-03-03T16:56:12.798046+0800
pg 2.3ca not scrubbed since 2024-03-04T06:04:37.872510+0800
pg 2.3c9 not scrubbed since 2024-03-04T22:47:57.180631+0800
pg 2.3c8 not scrubbed since 2024-03-04T19:16:57.384554+0800
pg 2.3c6 not scrubbed since 2024-03-04T19:37:51.044187+0800
pg 2.3c5 not scrubbed since 2024-03-04T11:43:06.678104+0800
pg 2.3c4 not scrubbed since 2024-03-04T20:15:50.165636+0800
pg 2.3c2 not scrubbed since 2024-03-04T19:22:16.316925+0800
pg 2.3c1 not scrubbed since 2024-03-05T00:03:16.565319+0800
pg 2.3bd not scrubbed since 2024-03-04T22:08:39.536820+0800
pg 2.3bc not scrubbed since 2024-03-04T23:19:13.508314+0800
pg 2.3ba not scrubbed since 2024-03-03T19:25:15.893331+0800
pg 2.3b9 not scrubbed since 2024-03-03T20:01:41.108637+0800
pg 2.3b8 not scrubbed since 2024-03-04T14:03:55.197692+0800
pg 2.3b7 not scrubbed since 2024-03-04T23:18:52.023791+0800
pg 2.3b6 not scrubbed since 2024-03-04T22:36:43.567151+0800
pg 2.3b5 not scrubbed since 2024-03-05T01:10:26.388716+0800
pg 2.3af not scrubbed since 2024-03-04T07:27:01.725128+0800
pg 2.3ae not scrubbed since 2024-03-05T03:55:10.714911+0800
pg 2.3ad not scrubbed since 2024-03-04T09:29:13.314556+0800
pg 2.3ab not scrubbed since 2024-03-03T07:44:35.921341+0800
553 more pgs...
root@px-sata-sg1-n1:~#
Are these numbers always there and always like this?
do you use some kind of automated snapshot mechanism and slow, rotating disks in your Ceph?