CephFS - filesystem is degraded

[Leo David]

Hello Guys,
I am running into a pretty big issue with PVE/Ceph, and i just can't figure out how to solve this..
I am using PVE 5.1, with vms running on external ceph rbd images. Ceph went into an outgate ( an osd flapped a lot, and finally went down ). Now i can't start vms, not CephFS can be accesible ( i am using cephfs mounted as a datastore too )
Here's my "ceph -s" output:

cluster:
id: 2806fcbd-4c9a-4805-a16a-10c01f3a9f32
health: HEALTH_ERR
1 filesystem is degraded
2 nearfull osd(s)
3 pool(s) nearfull
546309/7041370 objects misplaced (7.759%)
Reduced data availability: 323 pgs inactive, 1 pg peering
Degraded data redundancy: 224115/7041370 objects degraded (3.183%), 1253 pgs unclean, 47
5 pgs degraded, 466 pgs undersized
13 slow requests are blocked > 32 sec
too many PGs per OSD (298 > max 200)

services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3
mgr: ceph3(active), standbys: ceph1, ceph2
mds: cephfs-1/1/1 up {0=ceph3=up:replay}, 5 up:standby
osd: 37 osds: 36 up, 36 in; 1177 remapped pgs

data:
pools: 20 pools, 4712 pgs
objects: 2745k objects, 7965 GB
usage: 20644 GB used, 62068 GB / 82713 GB avail
pgs: 6.855% pgs not active
224115/7041370 objects degraded (3.183%)
546309/7041370 objects misplaced (7.759%)
3423 active+clean
599 active+remapped+backfill_wait
230 active+undersized+degraded
169 activating+remapped
96 active+undersized+degraded+remapped+backfill_wait
80 activating+undersized+degraded+remapped
58 undersized+degraded+peered
36 active+clean+remapped
8 activating+degraded
7 activating
2 active+remapped+backfilling
2 active+undersized+degraded+remapped+backfilling
1 remapped+peering
1 active+recovery_wait+degraded

As you can see, the mds is stuck in "replay" and mds logs are only showing:
2018-02-10 16:40:56.914050 7f3028ee11c0 0 ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable), process (unknown), pid 1818
2018-02-10 16:40:56.920917 7f3028ee11c0 0 pidfile_write: ignore empty --pid-file
2018-02-10 16:41:15.514658 7f3021d85700 1 mds.ceph3 handle_mds_map standby
2018-02-10 16:41:15.531339 7f3021d85700 1 mds.0.3606 handle_mds_map i am now mds.0.3606
2018-02-10 16:41:15.531349 7f3021d85700 1 mds.0.3606 handle_mds_map state change up:boot --> up:replay
2018-02-10 16:41:15.531359 7f3021d85700 1 mds.0.3606 replay_start
2018-02-10 16:41:15.531364 7f3021d85700 1 mds.0.3606 recovery set is
2018-02-10 16:41:15.531371 7f3021d85700 1 mds.0.3606 waiting for osdmap 181020 (which blacklists prior instance)

So waiting for osdmap, and that's it...
On the other hand none of the vms relying on rbd can be started.
Now, i have added 4 more osds to the cluster, and begun rebalancing as seen. But before that, cluster was rebalanced, not any misplaced or degraded objects, but still 308 inactive PGs.
I would really apreciate an advice, not sure how to proceed further.
Thank you very much, have a nice weekend !

Leo

 

[Jarek]

You have PGs stuck in activating state because of

too many PGs per OSD (298 > max 200)

Follow this
https://forum.proxmox.com/threads/c...ade-to-5-1-slow-requests-stuck-request.38586/
and then wait for recovery to complete.

 

[Leo David]

Thank you very, very much Jarek !!!
I've got rid of that warning as per that topic, now i'm waiting it to rebalance - still 245 pgs inactive.
I have reduced some pools to size 2 & min_size 2, just for sake of geeting it back online asap.
Also, do you think the "filesystem is degraded' error is related to the fact that mds's cant get osdmap because of inactive pgs and therefore stucked in "reply" ?
Is there a way to "force" the peering to get those inactive pgs back online before any other rebalancing operations?

 

[Jarek]

Also, do you think the "filesystem is degraded' error is related to the fact that mds's cant get osdmap because of inactive pgs and therefore stucked in "reply" ?

Probably yes.

Is there a way to "force" the peering to get those inactive pgs back online before any other rebalancing operations?

Please show output of

ceph health detail

 

[Leo David]

Here's "ceph healt detail" :
HEALTH_ERR 1 filesystem is degraded; 2 nearfull osd(s); 3 pool(s) nearfull; 237447/5623682 objects misplaced (4.222%); Reduced data availability: 213 pgs inactive, 73 pgs peering; Degraded data redundancy: 1120/5623682 objects degraded (0.020%), 524 pgs unclean, 15 pgs degraded, 10 pgs undersized; 250 slow requests are blocked > 32 sec; 173 stuck requests are blocked > 4096 sec
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
OSD_NEARFULL 2 nearfull osd(s)
osd.0 is near full
osd.11 is near full
POOL_NEARFULL 3 pool(s) nearfull
pool 'ssd' is nearfull
pool 'ssd-cache' is nearfull
pool 'ssd-rbd-cache-tier' is nearfull
OBJECT_MISPLACED 237447/5623682 objects misplaced (4.222%)
PG_AVAILABILITY Reduced data availability: 213 pgs inactive, 73 pgs peering
pg 9.134 is stuck inactive for 88082.909164, current state activating+remapped, last acting [8,15]
pg 9.135 is stuck inactive for 193035.258828, current state activating+undersized+degraded+remapped, last acting [11]
pg 9.149 is stuck inactive for 185582.005560, current state activating+undersized+degraded+remapped, last acting [7]
pg 9.14a is stuck inactive for 76272.170241, current state activating+remapped, last acting [4,15]
pg 9.17e is stuck inactive for 185583.100414, current state activating+remapped, last acting [11,7]
pg 9.367 is stuck inactive for 193035.264834, current state activating+remapped, last acting [4,0]
pg 9.36c is stuck inactive for 193035.258247, current state activating+undersized+degraded+remapped, last acting [11]
pg 9.374 is stuck inactive for 194248.193769, current state activating+remapped, last acting [4,7]
pg 9.380 is stuck inactive for 185583.076535, current state activating+remapped, last acting [11,15]
pg 9.39a is stuck inactive for 193035.257993, current state activating+remapped, last acting [11,7]
pg 9.39f is stuck inactive for 76251.309785, current state activating+remapped, last acting [15,11]
pg 9.3c3 is stuck inactive for 15523.123634, current state activating+remapped, last acting [4,15]
pg 9.3f1 is stuck inactive for 193035.263215, current state activating+undersized+degraded+remapped, last acting [11]
pg 9.3fb is stuck inactive for 87799.181717, current state activating+undersized+degraded+remapped, last acting [15]
pg 10.142 is stuck peering for 1212.959531, current state remapped+peering, last acting [14,6]
pg 10.149 is stuck inactive for 1229.860980, current state activating, last acting [6,2]
pg 10.153 is stuck peering for 1212.959116, current state remapped+peering, last acting [14,24]
pg 10.161 is stuck peering for 1206.466046, current state peering, last acting [14,37]
pg 10.174 is stuck peering for 1208.949370, current state remapped+peering, last acting [2,6]
pg 10.17f is stuck peering for 1206.469092, current state peering, last acting [14,29]
pg 10.18b is stuck inactive for 1282.632606, current state activating+remapped, last acting [6,2]
pg 10.18d is stuck peering for 1308.123512, current state peering, last acting [6,17]
pg 10.347 is stuck peering for 1229.471611, current state peering, last acting [6,39]
pg 10.34c is stuck peering for 1119.510074, current state peering, last acting [2,14]
pg 10.370 is stuck inactive for 1195.951338, current state activating+remapped, last acting [6,14]
pg 10.373 is stuck inactive for 6357.929456, current state undersized+degraded+remapped+backfilling+peered, last acting [6]
pg 10.389 is stuck peering for 1280.455049, current state remapped+peering, last acting [6,28]
pg 10.3ef is stuck peering for 1230.045685, current state peering, last acting [6,22]
pg 10.3fd is stuck peering for 1229.474229, current state peering, last acting [6,33]
pg 28.144 is stuck inactive for 87799.247921, current state activating+remapped, last acting [15,0]
pg 28.155 is stuck inactive for 768469.714625, current state activating+remapped, last acting [9,7]
pg 28.17e is stuck inactive for 87770.026467, current state activating+remapped, last acting [11,15]
pg 32.105 is stuck peering for 1230.044790, current state peering, last acting [6,34]
pg 32.11b is stuck peering for 1248.787032, current state peering, last acting [14,27]
pg 32.143 is stuck peering for 1230.048678, current state peering, last acting [6,16]
pg 32.14b is stuck peering for 1250.604073, current state peering, last acting [6,14]
pg 32.172 is stuck peering for 1212.894317, current state peering, last acting [14,31]
pg 33.107 is stuck peering for 8075.255396, current state remapped+peering, last acting [6,27]
pg 33.113 is stuck peering for 1089.309283, current state remapped+peering, last acting [2,28]
pg 33.156 is stuck inactive for 1077.696101, current state activating+remapped, last acting [28,3]
pg 33.16b is stuck peering for 1212.892791, current state peering, last acting [14,6]
pg 34.105 is stuck peering for 1229.931779, current state peering, last acting [2,5]
pg 34.179 is stuck peering for 1700.816166, current state remapped+peering, last acting [2,6]
pg 36.105 is stuck inactive for 185582.051830, current state activating+remapped, last acting [15,0]
pg 36.114 is stuck inactive for 76610.831050, current state activating+remapped, last acting [4,7]
pg 36.117 is stuck inactive for 185582.079644, current state activating+remapped, last acting [4,0]
pg 36.159 is stuck inactive for 76252.867550, current state activating+remapped, last acting [11,0]
pg 36.160 is stuck inactive for 15523.117211, current state activating+remapped, last acting [11,7]
pg 36.17b is stuck inactive for 193035.232014, current state activating+remapped, last acting [8,7]
pg 36.1ab is stuck inactive for 88070.881199, current state activating+remapped, last acting [15,7]
pg 36.1bc is stuck inactive for 185582.073684, current state activating+remapped, last acting [0,15]
PG_DEGRADED Degraded data redundancy: 1120/5623682 objects degraded (0.020%), 524 pgs unclean, 15 pgs degraded, 10 pgs undersized
pg 9.17e is stuck unclean for 185935.227421, current state activating+remapped, last acting [11,7]
pg 9.367 is stuck unclean for 193044.140278, current state activating+remapped, last acting [4,0]
pg 9.36c is stuck undersized for 76243.171662, current state activating+undersized+degraded+remapped, last acting [11]
pg 9.374 is stuck unclean for 238043.341279, current state activating+remapped, last acting [4,7]
pg 9.380 is stuck unclean for 185935.295970, current state activating+remapped, last acting [11,15]
pg 9.39a is stuck unclean for 238043.904634, current state activating+remapped, last acting [11,7]
pg 9.39f is stuck unclean for 193043.025725, current state activating+remapped, last acting [15,11]
pg 9.3c3 is stuck unclean for 15524.158866, current state activating+remapped, last acting [4,15]
pg 9.3f1 is stuck undersized for 76243.170503, current state activating+undersized+degraded+remapped, last acting [11]
pg 9.3fb is stuck undersized for 76243.153989, current state activating+undersized+degraded+remapped, last acting [15]
pg 10.182 is stuck unclean for 267205.468987, current state active+remapped+backfill_wait, last acting [5,30]
pg 10.184 is stuck unclean for 15681.244743, current state active+remapped+backfill_wait, last acting [26,37]
pg 10.18b is stuck unclean for 2160.826870, current state activating+remapped, last acting [6,2]
pg 10.18d is stuck unclean for 1308.123512, current state peering, last acting [6,17]
pg 10.341 is stuck unclean for 15780.332240, current state active+remapped+backfill_wait, last acting [32,6]
pg 10.343 is stuck unclean for 3588.820653, current state active+remapped+backfill_wait, last acting [3,23]
pg 10.344 is stuck unclean for 11138.032361, current state active+remapped+backfilling, last acting [13,17]
pg 10.347 is stuck unclean for 1229.471611, current state peering, last acting [6,39]
pg 10.34a is stuck unclean for 238527.582523, current state active+remapped+backfill_wait, last acting [39,28]
pg 10.34c is stuck unclean for 1119.510074, current state peering, last acting [2,14]
pg 10.363 is stuck unclean for 5173.846220, current state active+remapped+backfill_wait, last acting [31,13]
pg 10.367 is stuck unclean for 77067.529050, current state active+remapped+backfill_wait, last acting [14,31]
pg 10.370 is stuck unclean for 2160.821501, current state activating+remapped, last acting [6,14]
pg 10.371 is stuck unclean for 262917.353266, current state active+remapped+backfill_wait, last acting [2,26]
pg 10.373 is stuck undersized for 6106.844654, current state undersized+degraded+remapped+backfilling+peered, last acting [6]
pg 10.376 is stuck unclean for 6327.628162, current state active+remapped+backfill_wait, last acting [2,29]
pg 10.389 is stuck unclean for 238242.305332, current state remapped+peering, last acting [6,28]
pg 10.395 is stuck unclean for 78838.401465, current state active+remapped+backfill_wait, last acting [31,33]
pg 10.39a is stuck unclean for 189378.632761, current state active+remapped+backfill_wait, last acting [29,13]
pg 10.39c is stuck unclean for 79373.341771, current state active+remapped+backfill_wait, last acting [17,33]
pg 10.3a3 is stuck unclean for 267615.567010, current state active+remapped+backfill_wait, last acting [3,30]
pg 10.3aa is stuck unclean for 2202.404738, current state active+remapped+backfill_wait, last acting [29,33]
pg 10.3b1 is stuck unclean for 193653.017066, current state active+remapped+backfill_wait, last acting [35,17]
pg 10.3ba is stuck unclean for 6884.791740, current state active+remapped+backfilling, last acting [29,37]
pg 10.3c9 is stuck unclean for 130080.497688, current state active+remapped+backfill_wait, last acting [13,35]
pg 10.3cc is stuck unclean for 1095.183988, current state active+remapped+backfill_wait, last acting [13,2]
pg 10.3d5 is stuck unclean for 2202.394459, current state active+remapped+backfilling, last acting [32,39]
pg 10.3d9 is stuck unclean for 6327.342727, current state active+remapped+backfill_wait, last acting [3,32]
pg 10.3ea is stuck unclean for 2197.167908, current state active+remapped+backfill_wait, last acting [2,16]
pg 10.3ef is stuck unclean for 1230.045685, current state peering, last acting [6,22]
pg 10.3fc is stuck unclean for 263321.062490, current state active+remapped+backfill_wait, last acting [28,33]
pg 10.3fd is stuck unclean for 1229.474229, current state peering, last acting [6,33]
pg 33.156 is stuck unclean for 57750.836658, current state activating+remapped, last acting [28,3]
pg 33.157 is stuck unclean for 23881.846991, current state active+remapped+backfill_wait, last acting [5,28]
pg 33.1a6 is stuck unclean for 1656.456079, current state active+remapped+backfill_wait, last acting [26,35]
pg 33.1af is stuck unclean for 238430.310991, current state active+remapped+backfill_wait, last acting [30,37]
pg 34.156 is stuck unclean for 204364.704313, current state active+remapped+backfill_wait, last acting [2,35]
pg 34.1a1 is stuck unclean for 79373.205017, current state active+remapped+backfill_wait, last acting [37,3]
pg 34.1aa is stuck unclean for 481319.435606, current state active+remapped+backfill_wait, last acting [3,37]
pg 36.1ab is stuck unclean for 88073.817920, current state activating+remapped, last acting [15,7]
pg 36.1bc is stuck unclean for 185582.709964, current state activating+remapped, last acting [0,15]
REQUEST_SLOW 250 slow requests are blocked > 32 sec
42 ops are blocked > 2097.15 sec
105 ops are blocked > 1048.58 sec
51 ops are blocked > 524.288 sec
25 ops are blocked > 262.144 sec
13 ops are blocked > 131.072 sec
9 ops are blocked > 65.536 sec
5 ops are blocked > 32.768 sec
osd.6 has blocked requests > 32.768 sec
osds 2,14 have blocked requests > 2097.15 sec
REQUEST_STUCK 173 stuck requests are blocked > 4096 sec
27 ops are blocked > 134218 sec
74 ops are blocked > 67108.9 sec
38 ops are blocked > 33554.4 sec
20 ops are blocked > 16777.2 sec
10 ops are blocked > 8388.61 sec
4 ops are blocked > 4194.3 sec
osd.0 has stuck requests > 16777.2 sec
osds 4,8,9,11,15 have stuck requests > 134218 sec

 

[Jarek]

Check what causing the peering problem:
http://docs.ceph.com/docs/master/ra...ting-pg/#placement-group-down-peering-failure

 

[Leo David]

Thank you !
After this day, don;t have any misplaced or degraded objects, but still having inactive pgs...
Here's ceph -s , and ceph health detail :

cluster:
id: 2806fcbd-4c9a-4805-a16a-10c01f3a9f32
health: HEALTH_WARN
1 filesystem is degraded
2 nearfull osd(s)
3 pool(s) nearfull
Reduced data availability: 146 pgs inactive
Degraded data redundancy: 146 pgs unclean, 2 pgs degraded, 2 pgs undersized
125 slow requests are blocked > 32 sec

services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3
mgr: ceph1(active), standbys: ceph2, ceph3
mds: cephfs-1/1/1 up {0=ceph1=up:replay}, 5 up:standby
osd: 35 osds: 35 up, 35 in; 146 remapped pgs
rgw: 1 daemon active

data:
pools: 20 pools, 4712 pgs
objects: 2745k objects, 7965 GB
usage: 16207 GB used, 64641 GB / 80849 GB avail
pgs: 3.098% pgs not active
4563 active+clean
144 activating+remapped
3 active+clean+scrubbing+deep
2 activating+undersized+degraded+remapped



HEALTH_WARN 1 filesystem is degraded; 2 nearfull osd(s); 3 pool(s) nearfull; Reduced data availability: 146 pgs inactive; Degraded data redundancy: 146 pgs unclean, 2 pgs degraded, 2 pgs undersized; 128 slow requests are blocked > 32 sec
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
OSD_NEARFULL 2 nearfull osd(s)
osd.0 is near full
osd.11 is near full
POOL_NEARFULL 3 pool(s) nearfull
pool 'ssd' is nearfull
pool 'ssd-cache' is nearfull
pool 'ssd-rbd-cache-tier' is nearfull
PG_AVAILABILITY Reduced data availability: 146 pgs inactive
pg 9.ee is stuck inactive for 102626.123983, current state activating+remapped, last acting [9,7]
pg 9.f1 is stuck inactive for 1193.857985, current state activating+remapped, last acting [15,7]
pg 9.fa is stuck inactive for 218178.607448, current state activating+remapped, last acting [4,0]
pg 9.fe is stuck inactive for 210726.399967, current state activating+remapped, last acting [11,15]
pg 9.103 is stuck inactive for 210739.610827, current state activating+remapped, last acting [7,0]
pg 9.104 is stuck inactive for 210739.015627, current state activating+remapped, last acting [9,7]
pg 9.11a is stuck inactive for 624.434796, current state activating+remapped, last acting [11,0]
pg 9.128 is stuck inactive for 112940.576951, current state activating+remapped, last acting [9,15]
pg 9.134 is stuck inactive for 113226.251556, current state activating+remapped, last acting [8,15]
pg 9.135 is stuck inactive for 218178.601220, current state activating+remapped, last acting [11,0]
pg 9.149 is stuck inactive for 210725.347952, current state activating+remapped, last acting [7,0]
pg 9.14a is stuck inactive for 1193.869481, current state activating+remapped, last acting [4,15]
pg 9.17e is stuck inactive for 210726.442806, current state activating+remapped, last acting [11,7]
pg 9.367 is stuck inactive for 218178.607225, current state activating+remapped, last acting [4,0]
pg 9.36c is stuck inactive for 218178.600639, current state activating+remapped, last acting [11,0]
pg 9.374 is stuck inactive for 219391.536161, current state activating+remapped, last acting [4,7]
pg 9.380 is stuck inactive for 210726.418927, current state activating+remapped, last acting [11,15]
pg 9.38f is stuck inactive for 1463.030972, current state activating+remapped, last acting [11,0]
pg 9.39a is stuck inactive for 218178.600384, current state activating+remapped, last acting [11,7]
pg 9.39f is stuck inactive for 1193.855600, current state activating+remapped, last acting [15,11]
pg 9.3c3 is stuck inactive for 1424.918955, current state activating+remapped, last acting [4,15]
pg 9.3ca is stuck inactive for 624.416282, current state activating+remapped, last acting [7,0]
pg 9.3cc is stuck inactive for 1463.028072, current state activating+remapped, last acting [9,0]
pg 9.3f1 is stuck inactive for 218178.605606, current state activating+remapped, last acting [11,0]
pg 9.3f3 is stuck inactive for 405.165697, current state activating+remapped, last acting [9,0]
pg 9.3fb is stuck inactive for 112942.524109, current state activating+remapped, last acting [15,0]
pg 28.ff is stuck inactive for 1193.880132, current state activating+remapped, last acting [11,0]
pg 28.116 is stuck inactive for 101417.642993, current state activating+remapped, last acting [4,0]
pg 28.133 is stuck inactive for 1193.889357, current state activating+remapped, last acting [0,7]
pg 28.134 is stuck inactive for 1432.492816, current state activating+undersized+degraded+remapped, last acting [15]
pg 28.13d is stuck inactive for 22101.394011, current state activating+remapped, last acting [8,7]
pg 28.144 is stuck inactive for 22101.420516, current state activating+remapped, last acting [15,0]
pg 28.155 is stuck inactive for 405.169897, current state activating+remapped, last acting [9,7]
pg 28.168 is stuck inactive for 624.422422, current state activating+remapped, last acting [8,0]
pg 28.17e is stuck inactive for 22101.378316, current state activating+remapped, last acting [11,15]
pg 36.d9 is stuck inactive for 405.178117, current state activating+remapped, last acting [4,7]
pg 36.105 is stuck inactive for 1393.486750, current state activating+remapped, last acting [15,0]
pg 36.114 is stuck inactive for 1193.871946, current state activating+remapped, last acting [4,7]
pg 36.117 is stuck inactive for 1393.492775, current state activating+remapped, last acting [4,0]
pg 36.129 is stuck inactive for 405.076725, current state activating+remapped, last acting [0,11]
pg 36.130 is stuck inactive for 624.426808, current state activating+remapped, last acting [8,0]
pg 36.131 is stuck inactive for 405.177552, current state activating+remapped, last acting [15,0]
pg 36.136 is stuck inactive for 101743.327120, current state activating+remapped, last acting [0,15]
pg 36.13e is stuck inactive for 1193.890073, current state activating+remapped, last acting [0,7]
pg 36.13f is stuck inactive for 1424.892730, current state activating+remapped, last acting [7,15]
pg 36.153 is stuck inactive for 405.173311, current state activating+remapped, last acting [4,0]
pg 36.159 is stuck inactive for 101396.209942, current state activating+remapped, last acting [11,0]
pg 36.160 is stuck inactive for 1393.426864, current state activating+remapped, last acting [11,7]
pg 36.17b is stuck inactive for 1393.481245, current state activating+remapped, last acting [8,7]
pg 36.1ab is stuck inactive for 22101.415712, current state activating+remapped, last acting [15,7]
pg 36.1bc is stuck inactive for 1393.510830, current state activating+remapped, last acting [0,15]
PG_DEGRADED Degraded data redundancy: 146 pgs unclean, 2 pgs degraded, 2 pgs undersized
pg 9.ee is stuck unclean for 263186.736601, current state activating+remapped, last acting [9,7]
pg 9.f1 is stuck unclean for 1232.075228, current state activating+remapped, last acting [15,7]
pg 9.fa is stuck unclean for 218187.482691, current state activating+remapped, last acting [4,0]
pg 9.fe is stuck unclean for 211078.638283, current state activating+remapped, last acting [11,15]
pg 9.103 is stuck unclean for 211078.576099, current state activating+remapped, last acting [7,0]
pg 9.104 is stuck unclean for 218187.446112, current state activating+remapped, last acting [9,7]
pg 9.11a is stuck unclean for 885.357762, current state activating+remapped, last acting [11,0]
pg 9.128 is stuck unclean for 218186.905347, current state activating+remapped, last acting [9,15]
pg 9.134 is stuck unclean for 218186.904735, current state activating+remapped, last acting [8,15]
pg 9.135 is stuck unclean for 263186.938380, current state activating+remapped, last acting [11,0]
pg 9.149 is stuck unclean for 210726.019333, current state activating+remapped, last acting [7,0]
pg 9.14a is stuck unclean for 1232.108681, current state activating+remapped, last acting [4,15]
pg 9.17e is stuck unclean for 211078.569813, current state activating+remapped, last acting [11,7]
pg 9.367 is stuck unclean for 218187.482669, current state activating+remapped, last acting [4,0]
pg 9.36c is stuck unclean for 218186.366552, current state activating+remapped, last acting [11,0]
pg 9.374 is stuck unclean for 263186.683670, current state activating+remapped, last acting [4,7]
pg 9.380 is stuck unclean for 211078.638361, current state activating+remapped, last acting [11,15]
pg 9.38f is stuck unclean for 129631.428616, current state activating+remapped, last acting [11,0]
pg 9.39a is stuck unclean for 263187.247026, current state activating+remapped, last acting [11,7]
pg 9.39f is stuck unclean for 1232.107587, current state activating+remapped, last acting [15,11]
pg 9.3c3 is stuck unclean for 1427.232275, current state activating+remapped, last acting [4,15]
pg 9.3ca is stuck unclean for 885.341385, current state activating+remapped, last acting [7,0]
pg 9.3cc is stuck unclean for 129795.556351, current state activating+remapped, last acting [9,0]
pg 9.3f1 is stuck unclean for 263310.102011, current state activating+remapped, last acting [11,0]
pg 9.3f3 is stuck unclean for 450.640274, current state activating+remapped, last acting [9,0]
pg 9.3fb is stuck unclean for 113249.680510, current state activating+remapped, last acting [15,0]
pg 28.ff is stuck unclean for 1232.116647, current state activating+remapped, last acting [11,0]
pg 28.116 is stuck unclean for 101753.086129, current state activating+remapped, last acting [4,0]
pg 28.133 is stuck unclean for 1232.079792, current state activating+remapped, last acting [0,7]
pg 28.134 is stuck undersized for 1422.827578, current state activating+undersized+degraded+remapped, last acting [15]
pg 28.13d is stuck unclean for 22118.071035, current state activating+remapped, last acting [8,7]
pg 28.144 is stuck unclean for 22118.118083, current state activating+remapped, last acting [15,0]
pg 28.155 is stuck unclean for 450.638796, current state activating+remapped, last acting [9,7]
pg 28.168 is stuck unclean for 884.748122, current state activating+remapped, last acting [8,0]
pg 28.17e is stuck unclean for 22118.106433, current state activating+remapped, last acting [11,15]
pg 36.d9 is stuck unclean for 450.639292, current state activating+remapped, last acting [4,7]
pg 36.105 is stuck unclean for 1418.458061, current state activating+remapped, last acting [15,0]
pg 36.114 is stuck unclean for 1227.587347, current state activating+remapped, last acting [4,7]
pg 36.117 is stuck unclean for 1418.458722, current state activating+remapped, last acting [4,0]
pg 36.129 is stuck unclean for 450.664938, current state activating+remapped, last acting [0,11]
pg 36.130 is stuck unclean for 884.753949, current state activating+remapped, last acting [8,0]
pg 36.131 is stuck unclean for 450.666741, current state activating+remapped, last acting [15,0]
pg 36.136 is stuck unclean for 101752.052148, current state activating+remapped, last acting [0,15]
pg 36.13e is stuck unclean for 1232.079795, current state activating+remapped, last acting [0,7]
pg 36.13f is stuck unclean for 1427.250237, current state activating+remapped, last acting [7,15]
pg 36.153 is stuck unclean for 450.665627, current state activating+remapped, last acting [4,0]
pg 36.159 is stuck unclean for 101742.260251, current state activating+remapped, last acting [11,0]
pg 36.160 is stuck unclean for 1461.940216, current state activating+remapped, last acting [11,7]
pg 36.17b is stuck unclean for 1418.448654, current state activating+remapped, last acting [8,7]
pg 36.1ab is stuck unclean for 22118.116503, current state activating+remapped, last acting [15,7]
pg 36.1bc is stuck unclean for 1418.468201, current state activating+remapped, last acting [0,15]
REQUEST_SLOW 128 slow requests are blocked > 32 sec
4 ops are blocked > 2097.15 sec
21 ops are blocked > 1048.58 sec
55 ops are blocked > 524.288 sec
26 ops are blocked > 262.144 sec
12 ops are blocked > 131.072 sec
7 ops are blocked > 65.536 sec
3 ops are blocked > 32.768 sec
osd.8 has blocked requests > 524.288 sec
osds 9,11 have blocked requests > 2097.15 sec

And also last lines of osd.8 logs:

2018-02-11 15:45:14.539988 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 30.863142 seconds old, received at 2018-02-11 15:44:43.676547: osd_op(client.91747622.1:17154 36.ffa0bf6 36:6fd05ff0:::rbd_data.579a042ae8944a.0000000000000420:head [read 0~4096] snapc 0=[] RETRY=6 ack+retry+read+known_if_redirected e186495) currently waiting for peered
2018-02-11 15:45:44.546252 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 60.871917 secs
2018-02-11 15:45:44.546285 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 60.871917 seconds old, received at 2018-02-11 15:44:43.674240: osd_op(mds.0.3616:2 9.20b 9.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force e186495) currently waiting for peered
2018-02-11 15:45:44.546297 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 60.869610 seconds old, received at 2018-02-11 15:44:43.676547: osd_op(client.91747622.1:17154 36.ffa0bf6 36:6fd05ff0:::rbd_data.579a042ae8944a.0000000000000420:head [read 0~4096] snapc 0=[] RETRY=6 ack+retry+read+known_if_redirected e186495) currently waiting for peered
2018-02-11 15:46:44.558848 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 120.884482 secs
2018-02-11 15:46:44.558886 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 120.884482 seconds old, received at 2018-02-11 15:44:43.674240: osd_op(mds.0.3616:2 9.20b 9.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force e186495) currently waiting for peered
2018-02-11 15:46:44.558898 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 120.882175 seconds old, received at 2018-02-11 15:44:43.676547: osd_op(client.91747622.1:17154 36.ffa0bf6 36:6fd05ff0:::rbd_data.579a042ae8944a.0000000000000420:head [read 0~4096] snapc 0=[] RETRY=6 ack+retry+read+known_if_redirected e186495) currently waiting for peered
2018-02-11 15:48:44.584033 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 240.909709 secs
2018-02-11 15:48:44.584079 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 240.909709 seconds old, received at 2018-02-11 15:44:43.674240: osd_op(mds.0.3616:2 9.20b 9.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force e186495) currently waiting for peered
2018-02-11 15:48:44.584090 7f2f3f4c8700 0 log_channel(cluster) log [WRN] : slow request 240.907402 seconds old, received at 2018-02-11 15:44:43.676547: osd_op(client.91747622.1:17154 36.ffa0bf6 36:6fd05ff0:::rbd_data.579a042ae8944a.0000000000000420:head [read 0~4096] snapc 0=[] RETRY=6 ack+retry+read+known_if_redirected e186495) currently waiting for peered

I just can't understand why those guys are not activating the pgs, and "currently waiting for peered"....
Are there any chances for the cluster to heal itself and peer, activate pgs one by one overnight ( assuming no other bad events) ?

 

[Jarek]

How long you ran this cluster with HEALTH_WARN state?
With size=2 you agree with some data loss.

 

[Leo David]

Hi,
The WARN state showed up 3 days ago at the moment of outage. Now, CephFS eventually started. I have one cache tiering ssd pool that makes the ssd osds pretty dense regarding pgs, and after adding:
mon_max_pg_per_osd = 1500
osd_max_pg_per_osd_hard_ratio = 10
mon osd full ratio = .98
mon osd nearfull ratio = .95

The number of inactive pgs decreased to 18... i have 7 1tb ssd osds, the most loaded holds 631 pgs, the less one holds 563 pgs ( going to add tommorow another 4 ssd osds ). The thing is that i don;t have unfound or lost objects reported by "rados df" i think the pgs/objects are still there, but for some reason those 18pgs just don't peer/activate. My biggaest concern is about the base images for vms, there are a resonally number of vms depending on them as COW...
Here's my ceph -s now, and ceph health detail.
And again, thank you very much Jarek for your help !!!


cluster:
id: 2806fcbd-4c9a-4805-a16a-10c01f3a9f32
health: HEALTH_ERR
10964/4973204 objects misplaced (0.220%)
Reduced data availability: 18 pgs inactive
Degraded data redundancy: 35 pgs unclean
25 slow requests are blocked > 32 sec
74 stuck requests are blocked > 4096 sec

services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3
mgr: ceph3(active), standbys: ceph2, ceph1
mds: cephfs-1/1/1 up {0=ceph5=up:active}, 5 up:standby
osd: 37 osds: 37 up, 37 in; 35 remapped pgs
rgw: 2 daemons active

data:
pools: 20 pools, 4712 pgs
objects: 2428k objects, 7680 GB
usage: 15617 GB used, 71754 GB / 87371 GB avail
pgs: 0.382% pgs not active
10964/4973204 objects misplaced (0.220%)
4677 active+clean
18 activating+remapped
17 active+remapped+backfilling

io:
client: 5780 kB/s wr, 0 op/s rd, 4 op/s wr
recovery: 5440 kB/s, 1 objects/s
cache: 3708 kB/s flush, 4760 kB/s evict


HEALTH_ERR 10870/4972978 objects misplaced (0.219%); Reduced data availability: 18 pgs inactive; Degraded data redundancy: 35 pgs unclean; 10 slow requests are blocked > 32 sec; 74 stuck requests are blocked > 4096 sec
OBJECT_MISPLACED 10870/4972978 objects misplaced (0.219%)
PG_AVAILABILITY Reduced data availability: 18 pgs inactive
pg 9.12 is stuck inactive for 259339.868545, current state activating+remapped, last acting [7,11]
pg 9.6d is stuck inactive for 259339.897348, current state activating+remapped, last acting [11,0]
pg 9.135 is stuck inactive for 259339.896282, current state activating+remapped, last acting [11,0]
pg 9.1b1 is stuck inactive for 2763.980570, current state activating+remapped, last acting [8,0]
pg 9.22e is stuck inactive for 254141.849760, current state activating+remapped, last acting [4,11]
pg 9.2c4 is stuck inactive for 260552.968669, current state activating+remapped, last acting [4,7]
pg 9.302 is stuck inactive for 259339.858575, current state activating+remapped, last acting [8,11]
pg 9.38f is stuck inactive for 2764.003366, current state activating+remapped, last acting [11,0]
pg 9.3cc is stuck inactive for 2764.029812, current state activating+remapped, last acting [9,0]
pg 28.37 is stuck inactive for 2763.817277, current state activating+remapped, last acting [7,0]
pg 28.96 is stuck inactive for 2764.011197, current state activating+remapped, last acting [9,0]
pg 28.1e7 is stuck inactive for 2764.047633, current state activating+remapped, last acting [0,11]
pg 36.27 is stuck inactive for 2764.035084, current state activating+remapped, last acting [4,0]
pg 36.52 is stuck inactive for 2763.819452, current state activating+remapped, last acting [7,11]
pg 36.9d is stuck inactive for 2763.828024, current state activating+remapped, last acting [7,0]
pg 36.a0 is stuck inactive for 2764.013668, current state activating+remapped, last acting [11,7]
pg 36.159 is stuck inactive for 2764.009412, current state activating+remapped, last acting [11,0]
pg 36.1d3 is stuck inactive for 2764.002707, current state activating+remapped, last acting [11,0]
PG_DEGRADED Degraded data redundancy: 35 pgs unclean
pg 9.12 is stuck unclean for 259348.745596, current state activating+remapped, last acting [7,11]
pg 9.6d is stuck unclean for 259347.662537, current state activating+remapped, last acting [11,0]
pg 9.135 is stuck unclean for 304348.233442, current state activating+remapped, last acting [11,0]
pg 9.1b1 is stuck unclean for 2819.712381, current state activating+remapped, last acting [8,0]
pg 9.22e is stuck unclean for 259347.663459, current state activating+remapped, last acting [4,11]
pg 9.2c4 is stuck unclean for 304470.828454, current state activating+remapped, last acting [4,7]
pg 9.302 is stuck unclean for 259349.150453, current state activating+remapped, last acting [8,11]
pg 9.38f is stuck unclean for 2819.699180, current state activating+remapped, last acting [11,0]
pg 9.3cc is stuck unclean for 2819.699308, current state activating+remapped, last acting [9,0]
pg 10.57 is stuck unclean for 25176.167867, current state active+remapped+backfilling, last acting [16,31]
pg 10.142 is stuck unclean for 328.089618, current state active+remapped+backfilling, last acting [39,26]
pg 10.22d is stuck unclean for 5844.465198, current state active+remapped+backfilling, last acting [16,29]
pg 10.27e is stuck unclean for 328.096042, current state active+remapped+backfilling, last acting [6,31]
pg 10.282 is stuck unclean for 15473.469828, current state active+remapped+backfilling, last acting [28,39]
pg 10.290 is stuck unclean for 328.095763, current state active+remapped+backfilling, last acting [2,26]
pg 10.2ac is stuck unclean for 24369.146326, current state active+remapped+backfilling, last acting [14,27]
pg 10.2b5 is stuck unclean for 76448.208980, current state active+remapped+backfilling, last acting [14,27]
pg 10.324 is stuck unclean for 68503.747349, current state active+remapped+backfilling, last acting [13,29]
pg 10.3ea is stuck unclean for 15473.868419, current state active+remapped+backfilling, last acting [2,31]
pg 10.3f4 is stuck unclean for 328.083199, current state active+remapped+backfilling, last acting [39,32]
pg 28.37 is stuck unclean for 2819.672919, current state activating+remapped, last acting [7,0]
pg 28.96 is stuck unclean for 2819.693995, current state activating+remapped, last acting [9,0]
pg 28.1e7 is stuck unclean for 2819.699780, current state activating+remapped, last acting [0,11]
pg 33.a4 is stuck unclean for 61879.642655, current state active+remapped+backfilling, last acting [14,32]
pg 33.167 is stuck unclean for 73061.753207, current state active+remapped+backfilling, last acting [14,27]
pg 34.6 is stuck unclean for 15526.952288, current state active+remapped+backfilling, last acting [14,27]
pg 34.92 is stuck unclean for 6229.325637, current state active+remapped+backfilling, last acting [16,30]
pg 34.df is stuck unclean for 6229.399850, current state active+remapped+backfilling, last acting [16,26]
pg 34.11b is stuck unclean for 5830.900206, current state active+remapped+backfilling, last acting [16,29]
pg 36.27 is stuck unclean for 2819.725601, current state activating+remapped, last acting [4,0]
pg 36.52 is stuck unclean for 2819.686241, current state activating+remapped, last acting [7,11]
pg 36.9d is stuck unclean for 2819.691758, current state activating+remapped, last acting [7,0]
pg 36.a0 is stuck unclean for 2765.024985, current state activating+remapped, last acting [11,7]
pg 36.159 is stuck unclean for 2802.978647, current state activating+remapped, last acting [11,0]
pg 36.1d3 is stuck unclean for 2819.699185, current state activating+remapped, last acting [11,0]
REQUEST_SLOW 10 slow requests are blocked > 32 sec
2 ops are blocked > 2097.15 sec
1 ops are blocked > 1048.58 sec
1 ops are blocked > 262.144 sec
6 ops are blocked > 32.768 sec
osd.14 has blocked requests > 32.768 sec
REQUEST_STUCK 74 stuck requests are blocked > 4096 sec
74 ops are blocked > 4194.3 sec
osds 4,7,8,9,11 have stuck requests > 4194.3 sec

 

[Jarek]

So now you have a problem with osd 0 and 11 (no space left on device?).
With

mon osd full ratio = .98
mon osd nearfull ratio = .95

you only disable the warning, it will not free space on your osds. Maybe reweight this osds will help.

 

[Leo David]

Here's "ceph osd df" :

ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
2 hdd 1.00000 1.00000 4658G 731G 3927G 15.69 0.88 303
3 hdd 1.00000 1.00000 4658G 796G 3862G 17.09 0.96 336
5 hdd 1.00000 1.00000 4658G 667G 3991G 14.32 0.80 303
6 hdd 1.00000 1.00000 4658G 679G 3978G 14.60 0.82 297
13 hdd 1.00000 1.00000 4658G 740G 3917G 15.91 0.89 317
14 hdd 1.00000 1.00000 4658G 689G 3969G 14.80 0.83 305
16 hdd 1.00000 1.00000 4658G 738G 3920G 15.84 0.89 313
17 hdd 1.00000 1.00000 4658G 712G 3945G 15.30 0.86 294
19 hdd 1.00000 1.00000 1863G 79297M 1786G 4.15 0.23 37
20 hdd 1.00000 1.00000 1863G 114G 1749G 6.15 0.35 41
21 hdd 1.00000 1.00000 1863G 95167M 1770G 4.99 0.28 38
22 hdd 1.00000 1.00000 1863G 109G 1754G 5.88 0.33 47
23 hdd 1.00000 1.00000 1863G 118G 1745G 6.38 0.36 51
24 hdd 1.00000 1.00000 1863G 80626M 1785G 4.22 0.24 42
25 hdd 1.00000 1.00000 1863G 131G 1731G 7.08 0.40 50
26 hdd 1.00000 1.00000 1863G 601G 1261G 32.29 1.81 260
27 hdd 0 1.00000 1863G 14456M 1849G 0.76 0.04 3
28 hdd 1.00000 1.00000 1863G 556G 1307G 29.84 1.67 239
29 hdd 1.00000 1.00000 1863G 528G 1335G 28.33 1.59 235
30 hdd 1.00000 1.00000 1863G 487G 1376G 26.13 1.47 222
31 hdd 1.00000 1.00000 1863G 530G 1333G 28.44 1.60 231
32 hdd 1.00000 1.00000 1863G 635G 1228G 34.10 1.91 251
33 hdd 1.00000 1.00000 1863G 685G 1178G 36.79 2.06 306
34 hdd 0 1.00000 1863G 4607M 1859G 0.24 0.01 0
35 hdd 1.00000 1.00000 1863G 600G 1263G 32.22 1.81 257
37 hdd 1.00000 1.00000 1863G 627G 1236G 33.66 1.89 265
38 hdd 0 1.00000 1863G 4230M 1859G 0.22 0.01 0
39 hdd 1.00000 1.00000 1863G 721G 1142G 38.72 2.17 285
0 ssd 1.00000 0.89999 932G 458G 473G 49.20 2.76 563
4 ssd 1.00000 1.00000 895G 395G 499G 44.16 2.48 588
11 ssd 1.00000 0.95001 895G 501G 393G 56.00 3.14 600
15 ssd 1.00000 0.95001 895G 446G 448G 49.93 2.80 567
7 ssd 1.00000 0.95001 895G 416G 479G 46.48 2.61 565
8 ssd 1.00000 1.00000 895G 441G 453G 49.30 2.77 631
9 ssd 1.00000 1.00000 895G 426G 468G 47.66 2.67 582
10 hdd 0 1.00000 4658G 4789M 4653G 0.10 0.01 0
36 hdd 0 1.00000 1863G 3944M 1860G 0.21 0.01 0
TOTAL 87371G 15572G 71799G 17.82

0,4,11,15,7,8,9 hold the cahe-tiering pool. I just can;t figure out what could prevent those 18 pgs to wake up . Here's ceph osd tree too:

-24 19.00000 root sata
-2 2.00000 host ceph1-smr-sata
2 hdd 1.00000 osd.2 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
-55 2.00000 host ceph2-smr-sata
5 hdd 1.00000 osd.5 up 1.00000 0
6 hdd 1.00000 osd.6 up 1.00000 1.00000
-56 2.00000 host ceph3-smr-sata
13 hdd 1.00000 osd.13 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
-57 2.00000 host ceph4-smr-sata
16 hdd 1.00000 osd.16 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
-22 1.00000 host ceph5-sata
19 hdd 1.00000 osd.19 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
21 hdd 1.00000 osd.21 up 1.00000 1.00000
22 hdd 1.00000 osd.22 up 1.00000 1.00000
23 hdd 1.00000 osd.23 up 1.00000 1.00000
24 hdd 1.00000 osd.24 up 1.00000 1.00000
25 hdd 1.00000 osd.25 up 1.00000 1.00000
-19 6.00000 host ceph6-sata
26 hdd 1.00000 osd.26 up 1.00000 1.00000
27 hdd 0 osd.27 up 1.00000 1.00000
28 hdd 1.00000 osd.28 up 1.00000 1.00000
29 hdd 1.00000 osd.29 up 1.00000 1.00000
30 hdd 1.00000 osd.30 up 1.00000 1.00000
31 hdd 1.00000 osd.31 up 1.00000 1.00000
32 hdd 1.00000 osd.32 up 1.00000 1.00000
-17 4.00000 host ceph7-sata
33 hdd 1.00000 osd.33 up 1.00000 1.00000
34 hdd 0 osd.34 up 1.00000 1.00000
35 hdd 1.00000 osd.35 up 1.00000 1.00000
37 hdd 1.00000 osd.37 up 1.00000 1.00000
38 hdd 0 osd.38 up 1.00000 1.00000
39 hdd 1.00000 osd.39 up 1.00000 1.00000
-11 7.00000 root ssd
-4 1.00000 host ceph1-ssd
0 ssd 1.00000 osd.0 up 0.89999 1.00000
-5 1.00000 host ceph2-ssd
4 ssd 1.00000 osd.4 up 1.00000 1.00000
-6 1.00000 host ceph3-ssd
11 ssd 1.00000 osd.11 up 0.95001 1.00000
-8 1.00000 host ceph4-ssd
15 ssd 1.00000 osd.15 up 0.95001 1.00000
-1 1.00000 host ceph5-ssd
7 ssd 1.00000 osd.7 up 0.95001 1.00000
-3 1.00000 host ceph6-ssd
8 ssd 1.00000 osd.8 up 1.00000 1.00000
-10 1.00000 host ceph7-ssd
9 ssd 1.00000 osd.9 up 1.00000 1.00000
10 hdd 0 osd.10 up 1.00000 1.00000
36 hdd 0 osd.36 up 1.00000 1.00000

10 and 36 are about to be taken out..

 

[Leo David]

I am also thinking to temporarily disable the cache tier by "ceph osd tier cache-mode cache-tier-pool forward" and so, have the osds's freed up a bit ( if that would by the root cause of " 18 activating+remapped" )
The main issue with the sata pool ( thus avoidin to disable cache-tier) is that are pretty bad for ceph those spinning osds - western digital consumer grade
What do you think ?

Also, on all ssd osds, the last log lines are:

2018-02-12 10:24:22.335936 7f9e59c0d700 0 log_channel(cluster) log [WRN] : slow request 1921.815017 seconds old, received at 2018-02-12 09:52:20.520848: osd_op(mds.0.3637:9059 9.2c4 9.634716c4 (undecoded) ondisk+retry+write+known_if_redirected+full_force e188256) currently waiting for peered

 

[Leo David]

Finally cluster UP !!!
First, i've issued "ceph osd repair" agains all ssd osds that contained stucked pgs, then i've noticed that each of those 18 pgs, had osd.7 as the secondary in the dataset. This osd had reweight 0.89, i've changet it to 1.0, and pgs begun to peer and activate.
Now, the overall is flapping between:

cluster:
id: 2806fcbd-4c9a-4805-a16a-10c01f3a9f32
health: HEALTH_ERR
Possible data damage: 2 pgs repair

and

cluster:
id: 2806fcbd-4c9a-4805-a16a-10c01f3a9f32
health: HEALTH_OK

It can service requests, the vms are started, CephFS seems happy.
I guess/hope this big scrubbing of osds will finnish at a point without braking anything back...
Thank you very much Jarek, your advices have pointed me to right directions on the root cause !

 

[udo]

Hi,
I have not read the whole thread, but some points you should rethink about this:

- All of your OSDs have an weight of one, but they don't have the same size!
The weight should represent the disksize in TB (like 3.64 for an 4TB disk).

- Like wrote by Jarek before: replica of 2 is dangerous!!

- Use pgcalc from the ceph-website - it's looks for me, that you have too much PGs for your OSD-count!


Udo