[SOLVED] CEPH Recovery Stopped

[Mr.BlueBear]

We had a crash of multiple servers in our CEPH cluster. Recovery ran all night, but this morning it stopped.
No more movement. Also, I have a warning that my pool is full (100%), but there is still full of room on the OSDs. Since I am quite new to Proxmox and CEPH, I would like to know how to safely proceed.

root@saskatoon07:~# ceph status
  cluster:
    id:     40927eb1-05bf-48e6-928d-90ff7fa16f2e
    health: HEALTH_ERR
            1 full osd(s)
            1 nearfull osd(s)
            1 pool(s) full
            226/1674954 objects misplaced (0.013%)
            Degraded data redundancy: 229/1674954 objects degraded (0.014%), 4 pgs unclean, 4 pgs degraded, 1 pg undersized
            Degraded data redundancy (low space): 1 pg backfill_toofull, 3 pgs recovery_toofull
  services:
    mon: 5 daemons, quorum saskatoon04,saskatoon05,saskatoon06,saskatoon07,saskatoon08
    mgr: saskatoon04(active), standbys: saskatoon06, saskatoon05, saskatoon07, saskatoon08
    osd: 14 osds: 14 up, 14 in; 1 remapped pgs
  data:
    pools:   1 pools, 512 pgs
    objects: 545k objects, 2174 GB
    usage:   6565 GB used, 13705 GB / 20271 GB avail
    pgs:     229/1674954 objects degraded (0.014%)
             226/1674954 objects misplaced (0.013%)
             508 active+clean
             3   active+recovery_toofull+degraded
             1   active+undersized+degraded+remapped+backfill_toofull

root@saskatoon07:~# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE  VAR  PGS
 0   hdd 1.81969  1.00000  1863G  598G  1264G 32.13 0.99 140
 1   hdd 1.81999  1.00000  1863G  558G  1305G 29.96 0.93 131
 2   hdd 1.81969  1.00000  1863G  547G  1315G 29.38 0.91 128
 3   hdd 1.81969  1.00000  1863G  657G  1206G 35.28 1.09 154
 4   hdd 1.81969  1.00000  1863G  538G  1324G 28.90 0.89 126
 5   hdd 1.81969  1.00000  1863G  611G  1252G 32.81 1.01 143
 8   hdd 1.81969  1.00000  1862G  441G  1421G 23.68 0.73 103
 9   hdd 1.81969  1.00000  1862G  526G  1336G 28.24 0.87 122
 6   ssd 1.20000  1.00000   372G  324G 49126M 87.12 2.69  76
 7   ssd 1.20000  1.00000   372G  278G 95979M 74.84 2.31  65
12   hdd 1.81969  1.00000  1862G  399G  1462G 21.47 0.66  94
13   hdd 1.81969  1.00000  1862G  418G  1444G 22.48 0.69  98
10   ssd 1.20000  1.00000   447G  411G 35946M 92.15 2.85  96
11   ssd 1.20000  1.00000   447G  252G   194G 56.50 1.74  59
                    TOTAL 20271G 6565G 13705G 32.39         
MIN/MAX VAR: 0.66/2.85  STDDEV: 25.77

 

[Ashley]

From your ceph osd df you have couple of OSD over the full limit, this will stop any further re balancing otherwise these may hit the 100% mark and stop I/O.

Do you have a cache layer infront of the hdd pool using the 4 SSDs?

 

[Mr.BlueBear]

From your ceph osd df you have couple of OSD over the full limit, this will stop any further re balancing otherwise these may hit the 100% mark and stop I/O.

Do you have a cache layer infront of the hdd pool using the 4 SSDs?

I did have one, but not anymore. I was converting the whole cluster to have the block.db of each HDD on 1 SSD instead. So OSDs 0 to 5 have there block.db on an SSD and OSDs 6 to 11 are not (block.db on same device). My end goal is to take out the last 4 SSDs and place the block.db of each remaining HDD (8, 9, 12 and 13) on its own SSD.

So, to get re-balancing going again, do I lower the weight of the SSDs or just remove them ?

 

[Ashley]

I would suggest marking the most full SSD OSD as out.

This will allow I/O to still hit the SSD (read) but CEPH will start to move the data from this SSD to the other OSD's, just stopping the OSD's will make any data on these SSD's unavailable.

Once the first SSD is completed you can continue with the next, that should then get you to a state where all recovery is done, and you can then fully remove the 4 OUT SSD OSD's.

 

[Mr.BlueBear]

Thank you Ashley,

I marked the most full SSD OSD as out and that started the recovery process once again.
Should I be worried about the 2 scrub errors or wait until the whole recovery process is completed ?

root@saskatoon07:~# ceph status
  cluster:
    id:     40927eb1-05bf-48e6-928d-90ff7fa16f2e
    health: HEALTH_ERR
            1 nearfull osd(s)
            1 pool(s) nearfull
            91002/1676049 objects misplaced (5.430%)
            2 scrub errors
            Possible data damage: 1 pg inconsistent
            Degraded data redundancy: 62/1676049 objects degraded (0.004%), 96 pgs unclean, 3 pgs degraded
            Degraded data redundancy (low space): 3 pgs recovery_toofull
  services:
    mon: 5 daemons, quorum saskatoon04,saskatoon05,saskatoon06,saskatoon07,saskatoon08
    mgr: saskatoon04(active), standbys: saskatoon06, saskatoon05, saskatoon07, saskatoon08
    osd: 14 osds: 14 up, 13 in; 96 remapped pgs
  data:
    pools:   1 pools, 512 pgs
    objects: 545k objects, 2175 GB
    usage:   6215 GB used, 13608 GB / 19824 GB avail
    pgs:     62/1676049 objects degraded (0.004%)
             91002/1676049 objects misplaced (5.430%)
             415 active+clean
             54  active+remapped+backfilling
             39  active+remapped+backfill_wait
             3   active+recovery_toofull+degraded+remapped
             1   active+clean+inconsistent
  io:
    client:   67275 B/s rd, 5938 kB/s wr, 6 op/s rd, 126 op/s wr
    recovery: 173 MB/s, 43 objects/s

 

[Ashley]

I would wait for all your data to move around and the repair to fully complete, then let a full set of deep scrubs rotate.

While an OSD is in repair state it won't be deep scrubbed, so could just be a "false" positive.

 

[Ashley]

Is your issue fully resolved now?

 

[Mr.BlueBear]

Yes. I did need to run a Repair on some PGs, but dispite that, everything went well and I was able to remove the SSD OSDs.
Thanks.