[SOLVED] 'ceph pg 55.0 query' not working

[RobFantini]

hello , today I upgraded to the latest ceph. after the first note was upgraded i noticed an inactive pg warning. I continued and finished the upgrade hoping the inactive pg would be fixed with a complete upgrade. But that was not the case.

following info from https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/ to debug:

:# ceph -s
  cluster:
    id:     220b9a53-4556-48e3-a73c-28deff665e45
    health: HEALTH_WARN
            Reduced data availability: 1 pg inactive

# ceph health detail
HEALTH_WARN Reduced data availability: 1 pg inactive
[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive
    pg 55.0 is stuck inactive for 97m, current state unknown, last acting []

then this is where i hit a bump:

# ceph pg 55.0  query
Couldn't parse JSON : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/bin/ceph", line 1326, in <module>
    retval = main()
  File "/usr/bin/ceph", line 1246, in main
    sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 993, in parse_json_funcsigs
    raise e
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 990, in parse_json_funcsigs
    overall = json.loads(s)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I have not run those commands in years, so am not sure if the ceph query command was broken prior to the upgrade.

Can someone check if the ceph pg query rks on their Quincy ceph system?

 

[aaron]

Hmm, I think it has more to do that the PG is unknown, so no data available. That you see the parsing error is not nice.

But why is the PG unknown is the main question. Did you change disks or CRUSH rules?

 

[RobFantini]

Hmm, I think it has more to do that the PG is unknown, so no data available. That you see the parsing error is not nice.

But why is the PG unknown is the main question. Did you change disks or CRUSH rules?

I have not changed crush rules since 5 years ago.

 

[aaron]

So, what about the disks?

What is the output of ceph pg dump_stuck?

 

[RobFantini]

# ceph pg dump_stuck
PG_STAT  STATE                                            UP          UP_PRIMARY  ACTING   ACTING_PRIMARY
48.7e    active+undersized+degraded+remapped+backfilling   [1,24,19]           1   [1,24]               1
48.7d    active+undersized+degraded+remapped+backfilling   [10,2,42]          10   [10,2]              10
48.7a    active+undersized+degraded+remapped+backfilling    [0,41,2]           0   [0,41]               0
48.79    active+undersized+degraded+remapped+backfilling  [20,12,26]          20  [20,12]              20
48.78    active+undersized+degraded+remapped+backfilling  [19,20,41]          19  [19,20]              19
48.77    active+undersized+degraded+remapped+backfilling  [40,36,20]          40  [40,36]              40
48.76    active+undersized+degraded+remapped+backfilling   [16,38,0]          16  [16,38]              16
48.75    active+undersized+degraded+remapped+backfilling   [39,24,4]          39  [39,24]              39
48.74    active+undersized+degraded+remapped+backfilling  [12,16,14]          12  [12,16]              12
48.73    active+undersized+degraded+remapped+backfilling    [9,19,4]           9   [9,19]               9
48.72    active+undersized+degraded+remapped+backfilling  [10,21,41]          10  [10,21]              10
48.70    active+undersized+degraded+remapped+backfilling  [16,34,15]          16  [16,34]              16
48.6f    active+undersized+degraded+remapped+backfilling   [37,2,38]          37   [37,2]              37
48.12    active+undersized+degraded+remapped+backfilling    [1,8,39]           1    [1,8]               1
48.6d    active+undersized+degraded+remapped+backfilling  [38,26,21]          38  [38,26]              38
48.f     active+undersized+degraded+remapped+backfilling  [15,39,36]          15  [15,39]              15
48.e     active+undersized+degraded+remapped+backfilling   [41,3,24]          41   [41,3]              41
48.33    active+undersized+degraded+remapped+backfilling  [36,21,42]          36  [36,21]              36
48.10    active+undersized+degraded+remapped+backfilling   [8,11,25]           8   [8,11]               8
48.6b    active+undersized+degraded+remapped+backfilling   [38,14,8]          38  [38,14]              38
48.1     active+undersized+degraded+remapped+backfilling   [36,15,1]          36  [36,15]              36
48.5c    active+undersized+degraded+remapped+backfilling   [3,35,38]           3   [3,35]               3
48.7     active+undersized+degraded+remapped+backfilling  [16,21,12]          16  [16,21]              16
55.0                                             unknown          []          -1       []              -1
48.9     active+undersized+degraded+remapped+backfilling  [40,14,37]          40  [40,14]              40
48.a     active+undersized+degraded+remapped+backfilling   [3,42,24]           3   [3,42]               3
48.2b    active+undersized+degraded+remapped+backfilling   [1,12,14]           1   [1,12]               1
48.38    active+undersized+degraded+remapped+backfilling  [38,16,21]          38  [38,16]              38
48.4e    active+undersized+degraded+remapped+backfilling  [16,15,42]          16  [16,15]              16
48.7f    active+undersized+degraded+remapped+backfilling  [20,10,37]          20  [20,10]              20
48.25    active+undersized+degraded+remapped+backfilling   [12,35,9]          12  [12,35]              12
48.60    active+undersized+degraded+remapped+backfilling  [25,40,21]          25  [25,40]              25
48.c     active+undersized+degraded+remapped+backfilling  [40,37,25]          40  [40,37]              40
48.47    active+undersized+degraded+remapped+backfilling    [3,4,42]           3    [3,4]               3
48.2f    active+undersized+degraded+remapped+backfilling   [35,38,8]          35  [35,38]              35
48.6a    active+undersized+degraded+remapped+backfilling  [25,42,37]          25  [25,42]              25
48.1f    active+undersized+degraded+remapped+backfilling    [1,36,2]           1   [1,36]               1
48.5a    active+undersized+degraded+remapped+backfilling   [40,2,26]          40   [40,2]              40
48.22    active+undersized+degraded+remapped+backfilling  [25,12,19]          25  [25,12]              25
48.2e    active+undersized+degraded+remapped+backfilling  [10,16,14]          10  [10,16]              10
48.1e    active+undersized+degraded+remapped+backfilling  [26,40,34]          26  [26,40]              26
48.59    active+undersized+degraded+remapped+backfilling  [38,34,24]          38  [38,34]              38
48.21    active+undersized+degraded+remapped+backfilling   [12,9,10]          12   [12,9]              12
48.29    active+undersized+degraded+remapped+backfilling    [2,1,41]           2    [2,1]               2
48.5     active+undersized+degraded+remapped+backfilling  [21,41,35]          21  [21,41]              21
48.40    active+undersized+degraded+remapped+backfilling   [42,1,19]          42   [42,1]              42
48.28    active+undersized+degraded+remapped+backfilling    [12,3,2]          12   [12,3]              12
48.27    active+undersized+degraded+remapped+backfilling   [39,9,19]          39   [39,9]              39
48.62    active+undersized+degraded+remapped+backfilling   [0,40,20]           0   [0,40]               0
48.3     active+undersized+degraded+remapped+backfilling  [16,40,25]          16  [16,40]              16
48.5e    active+undersized+degraded+remapped+backfilling  [19,14,34]          19  [19,14]              19
48.26    active+undersized+degraded+remapped+backfilling   [14,8,41]          14   [14,8]              14
48.0     active+undersized+degraded+remapped+backfilling  [39,24,25]          39  [39,24]              39
48.23    active+undersized+degraded+remapped+backfilling   [1,36,21]           1   [1,36]               1
48.14    active+undersized+degraded+remapped+backfilling  [25,39,24]          25  [25,39]              25
48.15    active+undersized+degraded+remapped+backfilling   [16,4,40]          16   [16,4]              16
48.50    active+undersized+degraded+remapped+backfilling     [2,8,1]           2    [2,8]               2
48.17    active+undersized+degraded+remapped+backfilling  [25,26,14]          25  [25,26]              25
48.52    active+undersized+degraded+remapped+backfilling   [9,10,26]           9   [9,10]               9
48.3b    active+undersized+degraded+remapped+backfilling    [1,26,9]           1   [1,26]               1
48.3c    active+undersized+degraded+remapped+backfilling  [10,20,36]          10  [10,20]              10
48.3f    active+undersized+degraded+remapped+backfilling   [19,1,20]          19   [19,1]              19
48.43    active+undersized+degraded+remapped+backfilling   [12,14,3]          12  [12,14]              12
48.46    active+undersized+degraded+remapped+backfilling  [35,38,20]          35  [35,38]              35
48.48    active+undersized+degraded+remapped+backfilling   [37,39,1]          37  [37,39]              37
48.4c    active+undersized+degraded+remapped+backfilling  [25,41,21]          25  [25,41]              25
48.53    active+undersized+degraded+remapped+backfilling   [0,14,42]           0   [0,14]               0
48.57    active+undersized+degraded+remapped+backfilling  [15,42,34]          15  [15,42]              15
48.5f    active+undersized+degraded+remapped+backfilling    [39,2,0]          39   [39,2]              39
48.67    active+undersized+degraded+remapped+backfilling   [38,4,14]          38   [38,4]              38
48.68    active+undersized+degraded+remapped+backfilling  [41,21,37]          41  [41,21]              41
48.6e    active+undersized+degraded+remapped+backfilling   [24,19,1]          24  [24,19]              24
ok

 

[aaron]

Ok, no info at all. The question is, to which pool the PG belongs to.

You can run ceph osd pool ls detail. The pool number and the name will be listet right at the beginning of each line. The pool that the PG belongs to is "55".

Does a pool with that numerical ID exist? If so, is it one that you need?

And again, what happened that the PG became unknown? Did you have to replace some disks / OSDs?

 

[RobFantini]

And again, what happened that the PG became unknown? Did you have to replace some disks / OSDs?

i did not replace the disks, the only thing that changed was to upgrade ceph software.

we get emails from zabbix if ceph issues, and only got an email around the time of the upgrade .

Also I looked at the ceph section of the pve web page before upgrading and all was OK.

 

[RobFantini]

# ceph osd pool ls detail
pool 48 'nvme-4tb' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 300736 lfor 0/294914/294912 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 55 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 300877 flags hashpspool,creating stripe_width 0 pg_num_min 1 application mgr

 

[RobFantini]

# ceph osd pool ls detail
pool 48 'nvme-4tb' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 300736 lfor 0/294914/294912 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 55 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 300877 flags hashpspool,creating stripe_width 0 pg_num_min 1 application mgr

I did not create the .mgr pool. I suspect that got created during the upgrade. there was something in release notes about mgr ....

 

[aaron]

Hmm ok, then the problem shouldn't be too severe. The .mgr pool used to be the device_health_metrics.
You could try to delete the pool.
Hopefully that will also let Ceph forget about that PG:
ceph pg 55.0 mark_unfound_lost delete

Then you can recreate the pool. Check out this answer on how to get Ceph to do so. https://forum.proxmox.com/threads/ceph-pool-created-using-mgr.120226/#post-523686

 

[RobFantini]

trying to delete has this issue:

# ceph pg 55.0 mark_unfound_lost delete
Couldn't parse JSON : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/bin/ceph", line 1326, in <module>
    retval = main()
  File "/usr/bin/ceph", line 1246, in main
    sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 993, in parse_json_funcsigs
    raise e
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 990, in parse_json_funcsigs
    overall = json.loads(s)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

 

[RobFantini]

I was able to delete .mgr pool from the pve web page.

now I'll follow the link above on recreating the .mgr pool.

thank you for the help.

 

[RobFantini]

I was able to delete .mgr using pve web page.

after that the original warning went away:

 ceph -s
  cluster:
    id:     220b9a53-4556-48e3-a73c-28deff665e45
    health: HEALTH_WARN
            1 mgr modules have recently crashed
 
  services:
    mon: 3 daemons, quorum pve15,pve11,pve4 (age 14m)
    mgr: pve11(active, since 27s), standbys: pve15
    osd: 35 osds: 35 up (since 14m), 35 in (since 14m)
 
  data:
    pools:   2 pools, 129 pgs
    objects: 1.64M objects, 6.1 TiB
    usage:   18 TiB used, 109 TiB / 127 TiB avail
    pgs:     0.775% pgs unknown
             128 active+clean
             1   unknown
 
  io:
    client:   617 KiB/s rd, 5.6 MiB/s wr, 22 op/s rd, 404 op/s wr

then I added back the manager. 'Reduced data availability: 1 pg inactive' is back:

# ceph -s
  cluster:
    id:     220b9a53-4556-48e3-a73c-28deff665e45
    health: HEALTH_WARN
            Reduced data availability: 1 pg inactive
            2 mgr modules have recently crashed
 
  services:
    mon: 3 daemons, quorum pve15,pve11,pve4 (age 18m)
    mgr: pve11(active, since 4m), standbys: pve4, pve15
    osd: 35 osds: 35 up (since 18m), 35 in (since 18m)
 
  data:
    pools:   2 pools, 129 pgs
    objects: 1.64M objects, 6.1 TiB
    usage:   18 TiB used, 109 TiB / 127 TiB avail
    pgs:     0.775% pgs unknown
             128 active+clean
             1   unknown
 
  io:
    client:   19 KiB/s rd, 2.4 MiB/s wr, 1 op/s rd, 148 op/s wr

using : ceph crash archive-all , cleared '2 mgr modules have recently crashed' from ceph-s

 

[RobFantini]

so ceph health still has the original warning:

# ceph -s
  cluster:
    id:     220b9a53-4556-48e3-a73c-28deff665e45
    health: HEALTH_WARN
            Reduced data availability: 1 pg inactive
 
  services:
    mon: 3 daemons, quorum pve15,pve11,pve4 (age 6h)
    mgr: pve11(active, since 6h), standbys: pve4, pve15
    osd: 35 osds: 35 up (since 6h), 35 in (since 6h)
 
  data:
    pools:   2 pools, 129 pgs
    objects: 1.64M objects, 6.1 TiB
    usage:   18 TiB used, 109 TiB / 127 TiB avail
    pgs:     0.775% pgs unknown
             128 active+clean
             1   unknown
 
  io:
    client:   9.4 KiB/s rd, 1.5 MiB/s wr, 0 op/s rd, 90 op/s wr
 
root@pve11:[~]:# date
Wed 05 Apr 2023 07:01:27 PM EDT

 

[aaron]

Hmm... can you please post the output of the following commands? Something is weird...

• pveceph pool ls --noborder (make sure the terminal is wide enough or pipe it into a file)
• ceph osd df tree

 

[RobFantini]

# pveceph pool ls --noborder
Name     Size Min Size PG Num min. PG Num Optimal PG Num PG Autoscale Mode PG Autoscale Target Size PG Autoscale Target Ratio Crush Rule Name            %-Used Used
.mgr        3        2      1           1                on                                                                   s3700-800                       0 0
nvme-4tb    3        2    128                        128 on                                                                   nvme-4tb        0.168961733579636 19710022748442

# ceph osd df tree
ID   CLASS     WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME     
 -1            127.35405         -  127 TiB   18 TiB   18 TiB  369 MiB   44 GiB  109 TiB  14.11  1.00    -          root default 
-11                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve10
-14             25.47081         -   25 TiB  3.3 TiB  3.3 TiB   68 MiB  7.8 GiB   22 TiB  13.02  0.92    -              host pve11
 13  nvme-4tb    3.63869   1.00000  3.6 TiB  671 GiB  669 GiB   13 MiB  1.5 GiB  3.0 TiB  18.01  1.28   14      up          osd.13
 28  nvme-4tb    3.63869   1.00000  3.6 TiB  480 GiB  479 GiB  9.8 MiB  1.2 GiB  3.2 TiB  12.89  0.91   10      up          osd.28
 29  nvme-4tb    3.63869   1.00000  3.6 TiB  573 GiB  572 GiB   11 MiB  1.3 GiB  3.1 TiB  15.38  1.09   12      up          osd.29
 30  nvme-4tb    3.63869   1.00000  3.6 TiB  530 GiB  529 GiB   10 MiB  1.2 GiB  3.1 TiB  14.22  1.01   11      up          osd.30
 31  nvme-4tb    3.63869   1.00000  3.6 TiB  284 GiB  283 GiB  6.4 MiB  664 MiB  3.4 TiB   7.62  0.54    6      up          osd.31
 32  nvme-4tb    3.63869   1.00000  3.6 TiB  432 GiB  431 GiB  9.3 MiB  1.1 GiB  3.2 TiB  11.59  0.82    9      up          osd.32
 33  nvme-4tb    3.63869   1.00000  3.6 TiB  427 GiB  427 GiB  9.1 MiB  971 MiB  3.2 TiB  11.47  0.81    9      up          osd.33
-25                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve14
-21             25.47081         -   25 TiB  3.5 TiB  3.5 TiB   73 MiB  8.5 GiB   22 TiB  13.62  0.97    -              host pve15
  2  nvme-4tb    3.63869   1.00000  3.6 TiB  579 GiB  578 GiB   12 MiB  1.5 GiB  3.1 TiB  15.54  1.10   12      up          osd.2
  9  nvme-4tb    3.63869   1.00000  3.6 TiB  337 GiB  336 GiB  7.2 MiB  997 MiB  3.3 TiB   9.05  0.64    7      up          osd.9
 14  nvme-4tb    3.63869   1.00000  3.6 TiB  574 GiB  573 GiB   11 MiB  1.5 GiB  3.1 TiB  15.41  1.09   12      up          osd.14
 15  nvme-4tb    3.63869   1.00000  3.6 TiB  481 GiB  480 GiB  9.1 MiB  888 MiB  3.2 TiB  12.90  0.91   10      up          osd.15
 20  nvme-4tb    3.63869   1.00000  3.6 TiB  718 GiB  717 GiB   13 MiB  1.6 GiB  2.9 TiB  19.27  1.37   15      up          osd.20
 21  nvme-4tb    3.63869   1.00000  3.6 TiB  478 GiB  477 GiB   11 MiB  1.2 GiB  3.2 TiB  12.84  0.91   10      up          osd.21
 24  nvme-4tb    3.63869   1.00000  3.6 TiB  384 GiB  383 GiB  8.9 MiB  983 MiB  3.3 TiB  10.30  0.73    8      up          osd.24
-23             25.47081         -   25 TiB  3.3 TiB  3.3 TiB   67 MiB  8.0 GiB   22 TiB  13.05  0.92    -              host pve2
  3  nvme-4tb    3.63869   1.00000  3.6 TiB  572 GiB  571 GiB   11 MiB  1.3 GiB  3.1 TiB  15.36  1.09   12      up          osd.3
  8  nvme-4tb    3.63869   1.00000  3.6 TiB  191 GiB  191 GiB  4.4 MiB  526 MiB  3.5 TiB   5.13  0.36    4      up          osd.8
 16  nvme-4tb    3.63869   1.00000  3.6 TiB  527 GiB  526 GiB   10 MiB  1.3 GiB  3.1 TiB  14.15  1.00   11      up          osd.16
 19  nvme-4tb    3.63869   1.00000  3.6 TiB  527 GiB  526 GiB   12 MiB  1.2 GiB  3.1 TiB  14.16  1.00   11      up          osd.19
 26  nvme-4tb    3.63869   1.00000  3.6 TiB  770 GiB  768 GiB   14 MiB  1.7 GiB  2.9 TiB  20.67  1.46   16      up          osd.26
 36  nvme-4tb    3.63869   1.00000  3.6 TiB  479 GiB  478 GiB  9.2 MiB  1.1 GiB  3.2 TiB  12.85  0.91   10      up          osd.36
 37  nvme-4tb    3.63869   1.00000  3.6 TiB  335 GiB  335 GiB  6.8 MiB  807 MiB  3.3 TiB   9.00  0.64    7      up          osd.37
 -5                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve3
-34             25.47081         -   25 TiB  3.7 TiB  3.7 TiB   79 MiB  9.5 GiB   22 TiB  14.69  1.04    -              host pve4
  0  nvme-4tb    3.63869   1.00000  3.6 TiB  287 GiB  286 GiB  6.0 MiB  899 MiB  3.4 TiB   7.71  0.55    6      up          osd.0
  1  nvme-4tb    3.63869   1.00000  3.6 TiB  960 GiB  958 GiB   19 MiB  2.5 GiB  2.7 TiB  25.77  1.83   20      up          osd.1
  4  nvme-4tb    3.63869   1.00000  3.6 TiB  430 GiB  429 GiB  9.1 MiB  1.0 GiB  3.2 TiB  11.55  0.82    9      up          osd.4
 10  nvme-4tb    3.63869   1.00000  3.6 TiB  671 GiB  669 GiB   13 MiB  1.6 GiB  3.0 TiB  18.01  1.28   14      up          osd.10
 25  nvme-4tb    3.63869   1.00000  3.6 TiB  718 GiB  716 GiB   15 MiB  1.5 GiB  2.9 TiB  19.27  1.37   15      up          osd.25
 34  nvme-4tb    3.63869   1.00000  3.6 TiB  336 GiB  335 GiB  7.5 MiB  1.0 GiB  3.3 TiB   9.02  0.64    7      up          osd.34
 35  nvme-4tb    3.63869   1.00000  3.6 TiB  430 GiB  429 GiB  9.0 MiB  1.0 GiB  3.2 TiB  11.54  0.82    9      up          osd.35
-51                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve6
-56             25.47081         -   25 TiB  4.1 TiB  4.1 TiB   82 MiB   10 GiB   21 TiB  16.17  1.15    -              host pve7
 11  nvme-4tb    3.63869   1.00000  3.6 TiB  478 GiB  477 GiB  9.7 MiB  1.1 GiB  3.2 TiB  12.83  0.91   10      up          osd.11
 12  nvme-4tb    3.63869   1.00000  3.6 TiB  719 GiB  718 GiB   14 MiB  1.6 GiB  2.9 TiB  19.30  1.37   15      up          osd.12
 38  nvme-4tb    3.63869   1.00000  3.6 TiB  814 GiB  811 GiB   16 MiB  2.3 GiB  2.8 TiB  21.83  1.55   17      up          osd.38
 39  nvme-4tb    3.63869   1.00000  3.6 TiB  527 GiB  525 GiB   10 MiB  1.2 GiB  3.1 TiB  14.14  1.00   11      up          osd.39
 40  nvme-4tb    3.63869   1.00000  3.6 TiB  719 GiB  718 GiB   14 MiB  1.5 GiB  2.9 TiB  19.30  1.37   15      up          osd.40
 41  nvme-4tb    3.63869   1.00000  3.6 TiB  433 GiB  432 GiB  8.5 MiB  1.1 GiB  3.2 TiB  11.62  0.82    9      up          osd.41
 42  nvme-4tb    3.63869   1.00000  3.6 TiB  528 GiB  527 GiB   10 MiB  1.3 GiB  3.1 TiB  14.18  1.00   11      up          osd.42
-17                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host pve9
-36                    0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host sys8
                             TOTAL  127 TiB   18 TiB   18 TiB  369 MiB   44 GiB  109 TiB  14.11                                   
MIN/MAX VAR: 0.36/1.83  STDDEV: 4.41

 

[aaron]

Ah okay. I think we might be getting somewhere. The ".mgr" pool has the following crush rule assigned: "s3700-800".

I assume that you had OSDs with that device class (or similar) at some point? What does the rule look like if you check it in the CRUSH map? (Ceph -> Configuration -> right side of the screen, quite at the bottom).

If you do assign a rule for a specific device class to a pool and don't have any of those OSDs…, well Ceph has no chance to store it anywhere. Changing the assigned rule should help.

 

[pille99]

there is a ceph command to reset the warnings.
usually it should work automatically but once it didnt (for me) and i needed to reset the warnings/archive.
but i dont remember the command - maybe somebody here knows the command

 

[RobFantini]

Y

Ah okay. I think we might be getting somewhere. The ".mgr" pool has the following crush rule assigned: "s3700-800".

I assume that you had OSDs with that device class (or similar) at some point? What does the rule look like if you check it in the CRUSH map? (Ceph -> Configuration -> right side of the screen, quite at the bottom).

If you do assign a rule for a specific device class to a pool and don't have any of those OSDs…, well Ceph has no chance to store it anywhere. Changing the assigned rule should help.

Yes years ago I had set up a ceph rule like that , and we have since replaced the drives.

could you point me to documentation on changing the crush map rule?

 

[aaron]

You should be able to change the rule for the pool via the UI.

For device class specific rules, check this part of the Ceph docs: https://docs.ceph.com/en/latest/rados/operations/crush-map/#device-classes

With the ceph osd crush rule and further subcommands you can also remove the old rule after no pool is using it anymore.