Ceph cluster HEALTH_WARN 22% misplaced, 13% degraded

J

Jema

New Member
Jun 3, 2019
20
0
1
44
We recently started to add extra storage cause the main storage nodes were getting full and offloaded some. Only the cluster is not rebalancing correctly and OSD's are getting too full (85%). We added a new storage a few days ago but it's not getting filled at all and remains at 0.00 or 0.01% usage and I also notice the 'apply' and 'commit' latency is high compared to the others (latency (ms) is between 5-55ms on the Apply side and 0-15ms on the Commit side). Commit is always lower than apply.

Code: 
root@nvme02-nl-ams:~# ceph osd tree
ID WEIGHT   TYPE NAME              UP/DOWN REWEIGHT PRIMARY-AFFINITY
-2 17.58618 host nvme03-nl-ams
13  1.81360     osd.13                  up  1.00000          1.00000
14  1.81360     osd.14                  up  1.00000          1.00000
15  6.97949     osd.15                  up  1.00000          1.00000
 0  6.97949     osd.0                   up  1.00000          1.00000
-1 39.47479 root default
-3  5.76990     host hyp02-nl-ams
 4  0.71999         osd.4             down        0          1.00000
 5  0.71999         osd.5               up  1.00000          1.00000
 6  0.71999         osd.6               up  1.00000          1.00000
 7  0.71999         osd.7               up  1.00000          1.00000
 2  0.72249         osd.2               up  0.95001          1.00000
 3  0.72249         osd.3               up  0.95001          1.00000
16  0.72249         osd.16              up  0.95001          1.00000
18  0.72249         osd.18              up  0.95001          1.00000
-4  5.79956     host stor03-nl-ams
 8  1.44989         osd.8               up  0.70000          1.00000
 9  1.44989         osd.9               up  0.70000          1.00000
10  1.44989         osd.10              up  0.70000          1.00000
11  1.44989         osd.11            down        0          1.00000
-5 27.90533     host nvme02-nl-ams
 1  3.48659         osd.1               up  1.00000          1.00000
12  3.48659         osd.12              up  1.00000          1.00000
23  3.48659         osd.23              up  1.00000          1.00000
24  3.48659         osd.24              up  1.00000          1.00000
25  6.97949         osd.25              up  1.00000          1.00000
26  6.97949         osd.26              up  1.00000          1.00000

Code: 
root@nvme02-nl-ams:~# ceph -s
    cluster 1e3b6703-9bbb-4158-beee-e7ff2e6caee1
     health HEALTH_WARN
            762 pgs backfill_toofull
            1 pgs backfill_wait
            348 pgs degraded
            348 pgs stuck degraded
            789 pgs stuck unclean
            348 pgs stuck undersized
            348 pgs undersized
            recovery 496004/3743979 objects degraded (13.248%)
            recovery 1523562/3743979 objects misplaced (40.694%)
            10 near full osd(s)
     monmap e13: 3 mons at {2=10.3.5.5:6789/0,nvme02-nl-ams=10.3.5.200:6789/0,nvme03-nl-ams=10.3.5.202:6789/0}
            election epoch 398, quorum 0,1,2 2,nvme02-nl-ams,nvme03-nl-ams
     osdmap e28703: 22 osds: 20 up, 20 in; 788 remapped pgs
            flags nearfull,sortbitwise,require_jewel_osds
      pgmap v71139980: 1024 pgs, 1 pools, 4817 GB data, 1218 kobjects
            13320 GB used, 42895 GB / 56216 GB avail
            496004/3743979 objects degraded (13.248%)
            1523562/3743979 objects misplaced (40.694%)
                 422 active+remapped+backfill_toofull
                 339 active+undersized+degraded+remapped+backfill_toofull
                 235 active+clean
                  19 active+remapped
                   8 active+undersized+degraded
                   1 active+undersized+degraded+remapped+wait_backfill+backfill_toofull
  client io 285 MB/s rd, 24303 kB/s wr, 2095 op/s rd, 1427 op/s wr


Can anyone help me with this?
 
Last edited:
Code: 
root@nvme02-nl-ams:~# ceph health detail | less
HEALTH_WARN 762 pgs backfill_toofull; 2 pgs backfill_wait; 348 pgs degraded; 348 pgs stuck degraded; 789 pgs stuck unclean; 348 pgs stuck undersized; 348 pgs undersized; recovery 496014/3743994 objects degraded (13.248%); recovery 1523569/3743994 objects misplaced (40.694%); 10 near full osd(s)
pg 2.334 is stuck unclean for 258325.607572, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,8]
pg 2.333 is stuck unclean for 516.959927, current state active+remapped+backfill_toofull, last acting [6,9,24]
pg 2.332 is stuck unclean for 521.310979, current state active+remapped+backfill_toofull, last acting [3,9,23]
pg 2.331 is stuck unclean for 42502.186183, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,3]
pg 2.330 is stuck unclean for 42500.488177, current state active+undersized+degraded+remapped+backfill_toofull, last acting [6,25]
pg 2.32f is stuck unclean for 526.178170, current state active+remapped+backfill_toofull, last acting [7,9,25]
pg 2.32e is stuck unclean for 36423.605526, current state active+remapped+backfill_toofull, last acting [7,12,9]
pg 2.32d is stuck unclean for 36417.450620, current state active+remapped+backfill_toofull, last acting [26,10,16]
pg 2.32c is stuck unclean for 42500.605215, current state active+undersized+degraded+remapped+backfill_toofull, last acting [16,25]
pg 2.32b is stuck unclean for 513.602489, current state active+remapped+backfill_toofull, last acting [6,26,8]
pg 2.32a is stuck unclean for 113499.872899, current state active+remapped+backfill_toofull, last acting [12,10,25]
pg 2.329 is stuck unclean for 522.358527, current state active+remapped+backfill_toofull, last acting [12,8,18]
pg 2.328 is stuck unclean for 530.062357, current state active+remapped+backfill_toofull, last acting [5,8,26]
pg 2.327 is stuck unclean for 113502.580453, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24]
pg 2.326 is stuck unclean for 258323.439428, current state active+undersized+degraded+remapped+backfill_toofull, last acting [9,26]
pg 2.325 is stuck unclean for 36542.753323, current state active+remapped+backfill_toofull, last acting [1,18,8]
pg 2.324 is stuck unclean for 42499.976893, current state active+undersized+degraded+remapped+backfill_toofull, last acting [7,23]
pg 2.323 is stuck unclean for 513.180554, current state active+remapped+backfill_toofull, last acting [9,18,25]
pg 2.322 is stuck unclean for 42499.787587, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,3]
pg 2.321 is stuck unclean for 515.643742, current state active+remapped+backfill_toofull, last acting [24,5,9]
pg 2.320 is stuck unclean for 258326.415502, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1]
pg 2.31f is stuck unclean for 42500.474615, current state active+undersized+degraded+remapped+backfill_toofull, last acting [16,25]
pg 2.31e is stuck unclean for 518.402715, current state active+remapped+backfill_toofull, last acting [6,24,10]
pg 2.31d is stuck unclean for 113500.127835, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24]
pg 2.31c is stuck unclean for 513.282444, current state active+remapped+backfill_toofull, last acting [26,9,7]
pg 2.31b is stuck unclean for 36422.231831, current state active+remapped+backfill_toofull, last acting [9,16,25]
pg 2.31a is stuck unclean for 522.736002, current state active+remapped+backfill_toofull, last acting [2,8,23]
pg 2.319 is stuck unclean for 526.306379, current state active+remapped+backfill_toofull, last acting [7,10,12]
pg 2.318 is stuck unclean for 36422.130601, current state active+remapped+backfill_toofull, last acting [5,26,9]
pg 2.317 is stuck unclean for 635066.604791, current state active+remapped+backfill_toofull, last acting [3,9,25]
pg 2.316 is stuck unclean for 515.952004, current state active+remapped+backfill_toofull, last acting [7,9,23]
pg 2.315 is stuck unclean for 36417.089395, current state active+remapped+backfill_toofull, last acting [1,2,10]
pg 2.314 is stuck unclean for 113498.056656, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,25]
pg 2.313 is stuck unclean for 113499.872605, current state active+undersized+degraded+remapped+backfill_toofull, last acting [8,23]
pg 2.312 is stuck unclean for 515.013759, current state active+remapped+backfill_toofull, last acting [6,10,24]
pg 2.311 is stuck unclean for 258323.369825, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.310 is stuck unclean for 258322.556320, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1]
pg 2.30f is stuck unclean for 511.868308, current state active+remapped+backfill_toofull, last acting [26,10,16]
pg 2.30e is stuck unclean for 511.926776, current state active+remapped+backfill_toofull, last acting [2,10,25]
pg 2.30d is stuck unclean for 523.116773, current state active+remapped+backfill_toofull, last acting [10,3,25]
pg 2.30c is stuck unclean for 258324.403409, current state active+undersized+degraded+remapped+backfill_toofull, last acting [10,25]
pg 2.30b is stuck unclean for 36426.125381, current state active+remapped+backfill_toofull, last acting [2,10,25]
pg 2.30a is stuck unclean for 42505.232638, current state active+undersized+degraded+remapped+backfill_toofull, last acting [16,24]
pg 2.309 is stuck unclean for 516.740652, current state active+remapped+backfill_toofull, last acting [6,9,26]
pg 2.308 is stuck unclean for 516.263881, current state active+remapped+backfill_toofull, last acting [7,9,24]
pg 2.307 is stuck unclean for 42502.878580, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,18]
pg 2.306 is stuck unclean for 113499.562440, current state active+undersized+degraded+remapped+backfill_toofull, last acting [9,25]
pg 2.305 is stuck unclean for 514.110768, current state active+remapped+backfill_toofull, last acting [2,10,23]
pg 2.304 is stuck unclean for 42500.580327, current state active+undersized+degraded+remapped+backfill_toofull, last acting [7,23]
pg 2.349 is stuck unclean for 520.320225, current state active+remapped+backfill_toofull, last acting [5,10,25]
pg 2.34a is stuck unclean for 36416.108722, current state active+remapped+backfill_toofull, last acting [7,10,12]
pg 2.10 is stuck unclean for 113498.498038, current state active+remapped+backfill_toofull, last acting [8,25,1]
pg 2.34b is stuck unclean for 520.976188, current state active+remapped+backfill_toofull, last acting [2,10,12]
pg 2.34c is stuck unclean for 42507.626666, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,6]
pg 2.1a is stuck unclean for 36422.334751, current state active+remapped+backfill_toofull, last acting [24,18,10]
pg 2.34d is stuck unclean for 512.978470, current state active+remapped+backfill_toofull, last acting [9,16,25]
pg 2.17 is stuck unclean for 42506.433126, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,5]
pg 2.34e is stuck unclean for 113498.034856, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12]
pg 2.14 is stuck unclean for 635069.975141, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,10]
pg 2.34f is stuck unclean for 512.598215, current state active+remapped+backfill_toofull, last acting [2,9,25]
pg 2.19 is stuck unclean for 42499.687091, current state active+undersized+degraded+remapped+backfill_toofull, last acting [18,12]
pg 2.350 is stuck unclean for 514.620777, current state active+remapped+backfill_toofull, last acting [7,10,26]
pg 2.1e is stuck unclean for 113503.041716, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,25]
pg 2.351 is stuck unclean for 42500.479324, current state active+undersized+degraded+remapped+backfill_toofull, last acting [18,25]
pg 2.1b is stuck unclean for 42502.211107, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,5]
pg 2.352 is stuck unclean for 515.014430, current state active+remapped+backfill_toofull, last acting [5,9,24]
pg 2.18 is stuck unclean for 113498.740797, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.353 is stuck unclean for 36542.827059, current state active+remapped+backfill_toofull, last acting [6,8,26]
pg 2.1ba is stuck unclean for 634705.479473, current state active+undersized+degraded, last acting [12,5]
pg 2.1d is stuck unclean for 36422.317661, current state active+remapped+backfill_toofull, last acting [25,10,6]
pg 2.354 is stuck unclean for 113498.043406, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.22 is stuck unclean for 36426.125828, current state active+remapped+backfill_toofull, last acting [12,2,9]
pg 2.355 is stuck unclean for 514.913589, current state active+remapped+backfill_toofull, last acting [10,18,25]
pg 2.27 is stuck unclean for 42500.565317, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,6]
pg 2.1bc is stuck unclean for 113498.193784, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,8]
pg 2.356 is stuck unclean for 635067.163648, current state active+remapped+backfill_toofull, last acting [9,25,26]
pg 2.1b9 is stuck unclean for 258322.868611, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,9]
pg 2.1c is stuck unclean for 42510.321527, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,6]
pg 2.357 is stuck unclean for 42499.934047, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,5]
pg 2.1be is stuck unclean for 635113.947278, current state active+remapped+backfill_toofull, last acting [3,25,9]
pg 2.21 is stuck unclean for 258325.413571, current state active+remapped+backfill_toofull, last acting [23,10,18]
pg 2.358 is stuck unclean for 37703.424167, current state active+remapped+backfill_toofull, last acting [1,2,9]
pg 2.1c3 is stuck unclean for 258326.520838, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,9]
pg 2.359 is stuck unclean for 518.361545, current state active+remapped+backfill_toofull, last acting [7,10,12]
pg 2.1c0 is stuck unclean for 36538.604521, current state active+remapped+backfill_toofull, last acting [25,6,8]
pg 2.35a is stuck unclean for 36538.260423, current state active+remapped+backfill_toofull, last acting [1,8,7]
pg 2.1bd is stuck unclean for 113499.829716, current state active+remapped+backfill_toofull, last acting [12,8,26]
pg 2.20 is stuck unclean for 36417.131113, current state active+remapped+backfill_toofull, last acting [1,6,10]
pg 2.35b is stuck unclean for 113504.519049, current state active+remapped+backfill_toofull, last acting [8,24,25]
pg 2.35c is stuck unclean for 42499.680378, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,24]
pg 2.1c7 is stuck unclean for 634611.870104, current state active+remapped+backfill_toofull, last acting [26,10,6]
pg 2.2a is stuck unclean for 42500.496040, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,16]
pg 2.35d is stuck unclean for 512.507871, current state active+remapped+backfill_toofull, last acting [26,7,10]
pg 2.35f is stuck unclean for 514.746648, current state active+remapped+backfill_toofull, last acting [3,9,26]
pg 2.29 is stuck unclean for 113498.039810, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24]
pg 2.360 is stuck unclean for 36415.694466, current state active+remapped+backfill_toofull, last acting [10,16,23]
pg 2.2e is stuck unclean for 36446.191804, current state active+remapped+backfill_toofull, last acting [24,9,18]
pg 2.361 is stuck unclean for 42499.887923, current state active+undersized+degraded+remapped+backfill_toofull, last acting [6,23]
pg 2.1c8 is stuck unclean for 42500.633511, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,18]
pg 2.362 is stuck unclean for 36415.484914, current state active+remapped+backfill_toofull, last acting [10,16,26]
pg 2.bd is stuck undersized for 502.837490, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25]
pg 2.b3 is stuck undersized for 501.602922, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,10]
pg 2.b2 is stuck undersized for 502.661745, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,5]
pg 2.b is stuck undersized for 7231.392146, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,3]
pg 2.af is stuck undersized for 502.711990, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25]
pg 2.a9 is stuck undersized for 503.686815, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.a6 is stuck undersized for 507.074249, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,6]
pg 2.a4 is stuck undersized for 505.797622, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,24]
pg 2.a0 is stuck undersized for 502.665194, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25]
pg 2.a is stuck undersized for 7231.389209, current state active+undersized+degraded+remapped+backfill_toofull, last acting [18,23]
pg 2.9d is stuck undersized for 503.656645, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,7]
pg 2.90 is stuck undersized for 7231.397040, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,2]
pg 2.8f is stuck undersized for 7231.400243, current state active+undersized+degraded, last acting [23,6]
pg 2.8c is stuck undersized for 7231.393316, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,18]
pg 2.1b7 is stuck undersized for 505.663274, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,8]
pg 2.19 is stuck undersized for 7231.396047, current state active+undersized+degraded+remapped+backfill_toofull, last acting [18,12]
pg 2.1b3 is stuck undersized for 503.634615, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,16]
pg 2.18 is stuck undersized for 7231.394613, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.1b is stuck undersized for 7231.374170, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,5]
pg 2.16 is stuck undersized for 7231.391961, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26]
pg 2.1ad is stuck undersized for 497.735944, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,10]
pg 2.14 is stuck undersized for 7181.583074, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,10]
pg 2.1a9 is stuck undersized for 502.658746, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,16]
pg 2.1a4 is stuck undersized for 502.705731, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,18]
pg 2.19e is stuck undersized for 496.465343, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,8]
pg 2.19b is stuck undersized for 500.646464, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,7]
pg 2.335 is stuck undersized for 491.076018, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,26]
pg 2.2 is stuck undersized for 86795.787615, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,10]
pg 2.19a is stuck undersized for 496.465484, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,25]
pg 2.0 is stuck undersized for 86795.854671, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,10]
pg 2.196 is stuck undersized for 501.894679, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,8]
pg 2.194 is stuck undersized for 496.629190, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,2]
pg 2.18c is stuck undersized for 500.647490, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,9]
pg 2.18b is stuck undersized for 496.647340, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,3]
pg 2.188 is stuck undersized for 506.435600, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,5]
pg 2.181 is stuck undersized for 506.224566, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,12]
pg 2.180 is stuck undersized for 505.662376, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,8]
pg 2.1aa is stuck undersized for 507.775937, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,5]
pg 2.17d is stuck undersized for 508.118777, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,3]
pg 2.17c is stuck undersized for 503.644191, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,3]
pg 2.172 is stuck undersized for 507.009917, current state active+undersized+degraded+remapped+backfill_toofull, last acting [16,25]
pg 2.16d is stuck undersized for 503.233178, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,18]
pg 2.16a is stuck undersized for 506.924889, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24,3]
pg 2.166 is stuck undersized for 508.580617, current state active+undersized+degraded+remapped+backfill_toofull, last acting [12,7]
pg 2.164 is stuck undersized for 505.757816, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,25]
pg 2.161 is stuck undersized for 501.294823, current state active+undersized+degraded+remapped+backfill_toofull, last acting [23,6]
pg 2.15c is stuck undersized for 505.803471, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,24]
pg 2.127 is stuck undersized for 507.514264, current state active+undersized+degraded+remapped+backfill_toofull, last acting [24]
pg 2.126 is stuck undersized for 503.694705, current state active+undersized+degraded+remapped+backfill_toofull, last acting [26,9]
pg 2.124 is stuck undersized for 509.401532, current state active+undersized+degraded, last acting [23,7]
pg 2.122 is stuck undersized for 502.652239, current state active+undersized+degraded+remapped+backfill_toofull, last acting [25,3]
pg 2.11d is active+undersized+degraded+remapped+backfill_toofull, acting [24]
pg 2.131 is active+undersized+degraded+remapped+backfill_toofull, acting [1,3]
pg 2.19c is active+undersized+degraded+remapped+backfill_toofull, acting [25,9]
pg 2.120 is active+undersized+degraded+remapped+backfill_toofull, acting [1]
pg 2.122 is active+undersized+degraded+remapped+backfill_toofull, acting [25,3]
pg 2.193 is active+remapped+backfill_toofull, acting [10,3,12]
pg 2.124 is active+undersized+degraded, acting [23,7]
pg 2.125 is active+remapped+backfill_toofull, acting [1,8,18]
pg 2.126 is active+undersized+degraded+remapped+backfill_toofull, acting [26,9]
pg 2.192 is active+remapped+backfill_toofull, acting [1,9,16]
pg 2.127 is active+undersized+degraded+remapped+backfill_toofull, acting [24]
pg 2.158 is active+remapped+backfill_toofull, acting [1,9,2]
pg 2.15b is active+remapped+backfill_toofull, acting [24,8,25]
pg 2.15c is active+undersized+degraded+remapped+backfill_toofull, acting [2,24]
pg 2.1a0 is active+remapped+backfill_toofull, acting [24,10,18]
pg 2.160 is active+remapped+backfill_toofull, acting [23,16,10]
pg 2.161 is active+undersized+degraded+remapped+backfill_toofull, acting [23,6]
pg 2.162 is active+remapped+backfill_toofull, acting [26,16,10]
pg 2.164 is active+undersized+degraded+remapped+backfill_toofull, acting [3,25]
pg 2.166 is active+undersized+degraded+remapped+backfill_toofull, acting [12,7]
pg 2.167 is active+remapped+backfill_toofull, acting [16,12,10]
pg 2.16a is active+undersized+degraded+remapped+backfill_toofull, acting [24,3]
pg 2.16d is active+undersized+degraded+remapped+backfill_toofull, acting [25,18]
pg 2.1a1 is active+remapped+backfill_toofull, acting [23,10,18]
pg 2.172 is active+undersized+degraded+remapped+backfill_toofull, acting [16,25]
pg 2.176 is active+remapped+backfill_toofull, acting [18,9,26]
pg 2.177 is active+remapped+backfill_toofull, acting [26,5,8]
pg 2.178 is active+remapped+backfill_toofull, acting [23,18,8]
pg 2.17c is active+undersized+degraded+remapped+backfill_toofull, acting [26,3]
pg 2.17d is active+undersized+degraded+remapped+backfill_toofull, acting [23,3]
pg 2.1aa is active+undersized+degraded+remapped+backfill_toofull, acting [1,5]
pg 2.180 is active+undersized+degraded+remapped+backfill_toofull, acting [23,8]
pg 2.181 is active+undersized+degraded+remapped+backfill_toofull, acting [3,12]
pg 2.182 is active+remapped+backfill_toofull, acting [26,9,18]
pg 2.184 is active+remapped+backfill_toofull, acting [23,10,7]
pg 2.186 is active+remapped+backfill_toofull, acting [9,7,24]
pg 2.188 is active+undersized+degraded+remapped+backfill_toofull, acting [12,5]
pg 2.18a is active+remapped+backfill_toofull, acting [24,6,10]
pg 2.18b is active+undersized+degraded+remapped+backfill_toofull, acting [24,3]
pg 2.18c is active+undersized+degraded+remapped+backfill_toofull, acting [23,9]
pg 2.18d is active+remapped+backfill_toofull, acting [25,16,8]
pg 2.18e is active+remapped+backfill_toofull, acting [26,9,2]
pg 2.1ab is active+remapped+backfill_toofull, acting [25,6,9]
pg 2.191 is active+remapped+backfill_toofull, acting [24,8,2]
pg 2.194 is active+undersized+degraded+remapped+backfill_toofull, acting [24,2]
pg 2.196 is active+undersized+degraded+remapped+backfill_toofull, acting [12,8]
pg 2.327 is active+undersized+degraded+remapped+backfill_toofull, acting [24]
pg 2.328 is active+remapped+backfill_toofull, acting [5,8,26]
pg 2.329 is active+remapped+backfill_toofull, acting [12,8,18]
pg 2.32a is active+remapped+backfill_toofull, acting [12,10,25]
pg 2.32b is active+remapped+backfill_toofull, acting [6,26,8]
pg 2.32c is active+undersized+degraded+remapped+backfill_toofull, acting [16,25]
pg 2.32d is active+remapped+backfill_toofull, acting [26,10,16]
pg 2.32e is active+remapped+backfill_toofull, acting [7,12,9]
pg 2.32f is active+remapped+backfill_toofull, acting [7,9,25]
pg 2.330 is active+undersized+degraded+remapped+backfill_toofull, acting [6,25]
pg 2.331 is active+undersized+degraded+remapped+backfill_toofull, acting [1,3]
pg 2.332 is active+remapped+backfill_toofull, acting [3,9,23]
pg 2.333 is active+remapped+backfill_toofull, acting [6,9,24]
pg 2.334 is active+undersized+degraded+remapped+backfill_toofull, acting [1,8]
recovery 496014/3743994 objects degraded (13.248%)
recovery 1523569/3743994 objects misplaced (40.694%)
osd.2 is near full at 85%
osd.3 is near full at 85%
osd.5 is near full at 85%
osd.6 is near full at 86%
osd.7 is near full at 86%
osd.8 is near full at 85%
osd.9 is near full at 85%
osd.10 is near full at 85%
osd.16 is near full at 85%
osd.18 is near full at 85%
 
Last edited:
Code: 
root@nvme02-nl-ams:~# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USE    AVAIL   %USE  VAR  PGS
13 1.81360  1.00000  1857G   128M   1856G  0.01    0   0
14 1.81360  1.00000  1857G   128M   1856G  0.01    0   0
15 6.97949  1.00000  7146G   127M   7146G  0.00    0   0
 0 6.97949  1.00000  7146G   128M   7146G  0.00    0   0
 4 0.71999        0      0      0       0     0    0   0
 5 0.71999  1.00000   739G   634G    105G 85.72 3.62 225
 6 0.71999  1.00000   739G   640G 101681M 86.58 3.65 218
 7 0.71999  1.00000   739G   638G    101G 86.25 3.64 208
 2 0.72249  0.95001   739G   633G    106G 85.62 3.61 210
 3 0.72249  0.95001   739G   633G    105G 85.68 3.62 206
16 0.72249  0.95001   739G   634G    105G 85.80 3.62 196
18 0.72249  0.95001   739G   634G    105G 85.77 3.62 195
 8 1.44989  0.70000  1484G  1271G    213G 85.64 3.61 420
 9 1.44989  0.70000  1484G  1272G    211G 85.72 3.62 467
10 1.44989  0.70000  1484G  1267G    217G 85.37 3.60 437
11 1.44989        0      0      0       0     0    0   0
 1 3.48659  1.00000  3570G   466G   3104G 13.05 0.55 153
12 3.48659  1.00000  3570G   674G   2895G 18.89 0.80 198
23 3.48659  1.00000  3570G   682G   2887G 19.11 0.81 200
24 3.48659  1.00000  3570G   677G   2893G 18.96 0.80 197
25 6.97949  1.00000  7146G  1295G   5851G 18.12 0.76 357
26 6.97949  1.00000  7146G  1263G   5883G 17.68 0.75 364
              TOTAL 56216G 13320G  42895G 23.69
MIN/MAX VAR: 0/3.65  STDDEV: 44.17
 
Last edited:
Update:

Found the issue why those 4 OSD's were not getting used. I created a crush name outside of the existing pool, so those OSD's were not being used by the existing pool (dumb I know!).

I increased the pg_num value to 1024 and same for pgp_num, marked 2 OSD's as out and now it's rebalancing. If you don't hear me anymore, all worked out well. So far it's looking good.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom