Partition on ceph corrupt

alexskysilk · Feb 20, 2024

with so few OSDs of such a mix of capacities, its probably easier to reweight manually.

try

ceph osd reweight 0 0.8
ceph osd reweight 8 0.8

once those osd get under 80% full, you can try to start fixing your broken pgs.

And once you get your data back, I'd really suggest you either redeploy ceph properly or not at all.

Orionis · Feb 21, 2024

hi,
Good new the nearfull osd as gone.

Code:

cluster:
    id:     2c042659-77b4-4303-8ecb-3f6a88cd7d54
    health: HEALTH_WARN
            Reduced data availability: 39 pgs inactive, 39 pgs incomplete
            41 pgs not deep-scrubbed in time
            41 pgs not scrubbed in time

  services:
    mon: 2 daemons, quorum bsg-galactica,bsg-pegasus (age 30m)
    mgr: bsg-galactica(active, since 2d)
    osd: 11 osds: 11 up (since 12h), 11 in (since 2d); 52 remapped pgs

  data:
    pools:   2 pools, 503 pgs
    objects: 1.86M objects, 7.1 TiB
    usage:   14 TiB used, 6.7 TiB / 21 TiB avail
    pgs:     7.753% pgs not active
             212180/3725860 objects misplaced (5.695%)
             412 active+clean
             50  active+remapped+backfill_wait
             39  incomplete
             1   active+clean+remapped
             1   active+remapped+backfilling

  io:
    recovery: 341 KiB/s, 0 objects/s

i can try rpair or i stand again ?

alexskysilk · Feb 21, 2024

Orionis said:
i can try rpair or i stand again ?

now you follow https://forum.proxmox.com/threads/partition-on-ceph-corrupt.140915/post-634449

Orionis · Feb 21, 2024

ok but i've not directory in /var/lib/ceph/osd/osd-5/

the directory osd-5 exist but no current.....

alexskysilk · Feb 21, 2024

Orionis said:
the directory osd-5 exist but no current.....

I have no idea what that means. per that post, you should have a list of broken pgs- just start repairing them.

Orionis · Feb 21, 2024

i see that in the end of your post

A situation with 2 replicas can be a bit different, Ceph might not be able to solve this conflict and the problem could persist. So a simple trick could be to chose the latest version of the object, set the noout flag on the cluster, stop the OSD that has a wrong version. Wait a bit, start the OSD again and unset the noout flag. The cluster should sync up the good version of the object to OSD that had a wrong version.

i can try this ?

alexskysilk · Feb 22, 2024

its your data, you get to make the decisions.

Orionis · Feb 22, 2024

yes of course

i fact in this :https://medium.com/opsops/recoverin...-3-pgs-inactive-3-pgs-incomplete-b97cbcb4b5a1

he said pg with no data can be recreate without lost data.
see:

Code:

PG     OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES      OMAP_BYTES*  OMAP_KEYS*  LOG   STATE       SINCE  VERSION      REPORTED   UP         ACTING     SCRUB_STAMP                      DEEP_SCRUB_STAMP               
3.12         0         0          0        0          0            0           0  3222  incomplete    42m  6326'114958  8886:1572    [0,8]p0    [0,8]p0  2022-08-05T17:27:23.540156+0200  2022-08-01T18:31:44.084229+0200
3.2c         0         0          0        0          0            0           0  2545  incomplete    42m  6326'110177  8886:2460    [5,0]p5    [5,0]p5  2022-08-01T09:53:35.617819+0200  2022-07-28T13:36:48.842499+0200
3.41         0         0          0        0          0            0           0  5597  incomplete    42m  5493'125593  8886:2488    [6,0]p6    [6,0]p6  2022-08-06T14:04:48.123228+0200  2022-07-31T12:50:22.933559+0200
3.48         0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:284  [10,5]p10  [10,5]p10  2022-08-06T00:19:41.137858+0200  2022-08-02T08:37:35.502820+0200
3.57         0         0          0        0          0            0           0  2117  incomplete    42m  6326'116951  8886:1614    [0,5]p0    [0,5]p0  2022-08-06T04:22:46.075679+0200  2022-08-02T22:00:30.906588+0200
3.65         0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:288  [10,5]p10  [10,5]p10  2022-11-26T09:58:17.790782+0100  2022-11-20T04:38:11.107513+0100
3.68         0         0          0        0          0            0           0  5839  incomplete    42m  6326'116256  8886:1580    [0,5]p0    [0,5]p0  2022-07-31T22:23:46.225507+0200  2022-07-26T23:13:26.879814+0200
3.70         0         0          0        0          0            0           0  1913  incomplete    42m  6329'126956  8886:2415   [5,10]p5   [5,10]p5  2022-11-27T00:42:13.122933+0100  2022-11-20T21:16:11.613272+0100
3.75         0         0          0        0          0            0           0     0  incomplete    42m          0'0  8886:1249    [1,5]p1    [1,5]p1  2022-11-26T05:17:33.120909+0100  2022-11-23T21:14:08.772646+0100
3.8e         0         0          0        0          0            0           0  4137  incomplete    42m  6326'119937  8886:2481    [5,0]p5    [5,0]p5  2022-11-26T12:26:19.057741+0100  2022-11-22T15:19:51.327080+0100
3.92         0         0          0        0          0            0           0  4570  incomplete    42m  6326'114733  8886:1598    [0,8]p0    [0,8]p0  2022-08-05T17:27:23.540156+0200  2022-08-01T18:31:44.084229+0200
3.ac         0         0          0        0          0            0           0  3950  incomplete    42m  6329'110281  8886:2459    [5,0]p5    [5,0]p5  2022-08-01T09:53:35.617819+0200  2022-07-28T13:36:48.842499+0200
3.c1         0         0          0        0          0            0           0  5580  incomplete    42m  5258'125576  8886:2494    [6,0]p6    [6,0]p6  2022-08-06T14:04:48.123228+0200  2022-07-31T12:50:22.933559+0200
3.c8         0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:285  [10,5]p10  [10,5]p10  2022-08-06T00:19:41.137858+0200  2022-08-02T08:37:35.502820+0200
3.d7         0         0          0        0          0            0           0  2684  incomplete    42m  6326'117494  8886:1583    [0,5]p0    [0,5]p0  2022-08-06T04:22:46.075679+0200  2022-08-02T22:00:30.906588+0200
3.e5         0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:288  [10,5]p10  [10,5]p10  2022-11-26T09:58:17.790782+0100  2022-11-20T04:38:11.107513+0100
3.e8         0         0          0        0          0            0           0  5875  incomplete    42m  6326'116292  8886:1599    [0,5]p0    [0,5]p0  2022-07-31T22:23:46.225507+0200  2022-07-26T23:13:26.879814+0200
3.f0         0         0          0        0          0            0           0  6243  incomplete    42m  6326'126883  8886:2413   [5,10]p5   [5,10]p5  2022-11-27T00:42:13.122933+0100  2022-11-20T21:16:11.613272+0100
3.f5         0         0          0        0          0            0           0     0  incomplete    42m          0'0  8886:1249    [1,5]p1    [1,5]p1  2022-11-26T05:17:33.120909+0100  2022-11-23T21:14:08.772646+0100
3.10e        0         0          0        0          0            0           0  5985  incomplete    42m  6329'119868  8886:2480    [5,0]p5    [5,0]p5  2022-11-26T12:26:19.057741+0100  2022-11-22T15:19:51.327080+0100
3.112        0         0          0        0          0            0           0  5151  incomplete    42m  6326'114962  8886:1567    [0,8]p0    [0,8]p0  2022-08-05T17:27:23.540156+0200  2022-08-01T18:31:44.084229+0200
3.12c        0         0          0        0          0            0           0  5314  incomplete    42m  6326'110052  8886:2457    [5,0]p5    [5,0]p5  2022-08-01T09:53:35.617819+0200  2022-07-28T13:36:48.842499+0200
3.141        0         0          0        0          0            0           0  1906  incomplete    42m  5258'125583  8886:2496    [6,0]p6    [6,0]p6  2022-08-06T14:04:48.123228+0200  2022-07-31T12:50:22.933559+0200
3.148        0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:285  [10,5]p10  [10,5]p10  2022-08-06T00:19:41.137858+0200  2022-08-02T08:37:35.502820+0200
3.157        0         0          0        0          0            0           0  5849  incomplete    42m  6326'116762  8886:1632    [0,5]p0    [0,5]p0  2022-08-06T04:22:46.075679+0200  2022-08-02T22:00:30.906588+0200
3.165        0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:287  [10,5]p10  [10,5]p10  2022-11-26T09:58:17.790782+0100  2022-11-20T04:38:11.107513+0100
3.168        0         0          0        0          0            0           0  3815  incomplete    42m  6326'116350  8886:1585    [0,5]p0    [0,5]p0  2022-07-31T22:23:46.225507+0200  2022-07-26T23:13:26.879814+0200
3.170        0         0          0        0          0            0           0  5845  incomplete    42m  6326'126339  8886:2417   [5,10]p5   [5,10]p5  2022-11-27T00:42:13.122933+0100  2022-11-20T21:16:11.613272+0100
3.175        0         0          0        0          0            0           0     0  incomplete    42m          0'0  8886:1249    [1,5]p1    [1,5]p1  2022-11-26T05:17:33.120909+0100  2022-11-23T21:14:08.772646+0100
3.18e       71         0          0        0  301989888            0           0  2017  incomplete    42m  6326'120235  8886:2481    [5,0]p5    [5,0]p5  2022-11-26T12:26:19.057741+0100  2022-11-22T15:19:51.327080+0100
3.192        0         0          0        0          0            0           0  5828  incomplete    42m  6326'114783  8886:1562    [0,8]p0    [0,8]p0  2022-08-05T17:27:23.540156+0200  2022-08-01T18:31:44.084229+0200
3.1ac        0         0          0        0          0            0           0  5837  incomplete    42m  6326'110065  8886:2459    [5,0]p5    [5,0]p5  2022-08-01T09:53:35.617819+0200  2022-07-28T13:36:48.842499+0200
3.1c1        0         0          0        0          0            0           0  5607  incomplete    42m  5493'125603  8886:2495    [6,0]p6    [6,0]p6  2022-08-06T14:04:48.123228+0200  2022-07-31T12:50:22.933559+0200
3.1c8        0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:285  [10,5]p10  [10,5]p10  2022-08-06T00:19:41.137858+0200  2022-08-02T08:37:35.502820+0200
3.1d7        0         0          0        0          0            0           0  5880  incomplete    42m  6326'116793  8886:1607    [0,5]p0    [0,5]p0  2022-08-06T04:22:46.075679+0200  2022-08-02T22:00:30.906588+0200
3.1e5        0         0          0        0          0            0           0     0  incomplete    42m          0'0   8886:288  [10,5]p10  [10,5]p10  2022-11-26T09:58:17.790782+0100  2022-11-20T04:38:11.107513+0100
3.1e8        0         0          0        0          0            0           0  2707  incomplete    42m  6326'116644  8886:1629    [0,5]p0    [0,5]p0  2022-07-31T22:23:46.225507+0200  2022-07-26T23:13:26.879814+0200
3.1f0      156         0          0        0  653565952            0           0  4809  incomplete    42m  6326'126623  8886:2415   [5,10]p5   [5,10]p5  2022-11-27T00:42:13.122933+0100  2022-11-20T21:16:11.613272+0100
3.1f5        0         0          0        0          0            0           0     0  incomplete    42m          0'0  8886:1249    [1,5]p1    [1,5]p1  2022-11-26T05:17:33.120909+0100  2022-11-23T21:14:08.772646+0100

* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details.

i have just 2 npg with data 3.1f0 and 3.18.

i have launch ceph pg reapir 3.18 but it seem nothing append. in this two pg there 900 Mo . but i dont know if i drestroy it i can acces on my files after or not.

and i don't know other way to recover it.

Orionis · Feb 22, 2024

if i try ceph pg 3.18 mark_unfound_lost revert what happen on the datas in ?

sg90 · Feb 22, 2024

Orionis said:
if i try ceph pg 3.18 mark_unfound_lost revert what happen on the datas in ?

It will ignore any differences between the two copy’s the the PG’s will bring the PG online but the data within it will be corrupt and/or missing.

But being you only have two copies of the data as was running replication of 2 you don’t have too many options.

Orionis · Mar 6, 2024

Hi it's worse now
after the end of rebalancing, i start to repair the pg like this
https://medium.com/opsops/recoverin...-3-pgs-inactive-3-pgs-incomplete-b97cbcb4b5a1

after a few time one of my osd became down . and i can't restart.

i bought a new HDD 4To for help the system and when i put them and create the osd it stay down ( it's new and empty)

i restart the two node and i have a third OSD won't start. in log the three osd have the same end . I post the last new osd log because it was samller.

Orionis · Mar 6, 2024

Sorry when i post the last message a fourth osd is now down

Search

Search

Partition on ceph corrupt

alexskysilk

Distinguished Member

Orionis

Member

alexskysilk

Distinguished Member

Orionis

Member

alexskysilk

Distinguished Member

Orionis

Member

alexskysilk

Distinguished Member

Orionis

Member

Orionis

Member

sg90

Renowned Member

Orionis

Member

Attachments

Orionis

Member