Ceph health

[9acca9]

Hi.
The ceph is not healthy... what should i do??


ceph health
HEALTH_ERR 2 backfillfull osd(s); 1 pool(s) backfillfull; 258055/1493085 objects misplaced (17.283%); Degraded data redundancy: 98917/1493085 objects degraded (6.625%), 82 pgs unclean, 51 pgs degraded, 51 pgs undersized; Degraded data redundancy (low space): 82 pgs backfill_toofull



ceph health detail
HEALTH_ERR 2 backfillfull osd(s); 1 pool(s) backfillfull; 258055/1493085 objects misplaced (17.283%); Degraded data redundancy: 98917/1493085 objects degraded (6.625%), 82 pgs unclean, 51 pgs degraded, 51 pgs undersized; Degraded data redundancy (low space): 82 pgs backfill_toofull
OSD_BACKFILLFULL 2 backfillfull osd(s)
osd.4 is backfill full
osd.5 is backfill full
POOL_BACKFILLFULL 1 pool(s) backfillfull
pool 'Ceph01' is backfillfull
OBJECT_MISPLACED 258055/1493085 objects misplaced (17.283%)
PG_DEGRADED Degraded data redundancy: 98917/1493085 objects degraded (6.625%), 82 pgs unclean, 51 pgs degraded, 51 pgs undersized
pg 4.7 is stuck unclean for 143536.150799, current state active+remapped+backfill_toofull, last acting [2,0,5]
pg 4.8 is stuck undersized for 143528.926713, current state active+undersized+degraded+remapped+backfill_toofull, last acting [4,3]
pg 4.a is stuck unclean for 165140.114863, current state active+remapped+backfill_toofull, last acting [5,1,2]
pg 4.e is stuck unclean for 166086.407037, current state active+remapped+backfill_toofull, last acting [2,1,4]
pg 4.f is stuck undersized for 143532.514652, current state active+undersized+degraded+remapped+backfill_toofull, last acting [0,2]
pg 4.10 is stuck undersized for 156565.078927, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,1]
pg 4.13 is stuck undersized for 143532.514563, current state active+undersized+degraded+remapped+backfill_toofull, last acting [0,4]
pg 4.19 is stuck undersized for 143440.469241, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,2]
pg 4.1a is stuck undersized for 143439.077524, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,0]
pg 4.1b is stuck unclean for 165141.969265, current state active+remapped+backfill_toofull, last acting [3,1,4]
pg 4.1f is stuck undersized for 143438.982673, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,1]
pg 4.21 is stuck unclean for 166363.982939, current state active+remapped+backfill_toofull, last acting [5,1,3]
pg 4.24 is stuck unclean for 143536.251995, current state active+remapped+backfill_toofull, last acting [1,2,5]
pg 4.28 is stuck undersized for 143532.625919, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,0]
pg 4.29 is stuck undersized for 143532.623885, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,1]
pg 4.2b is stuck unclean for 165142.078857, current state active+remapped+backfill_toofull, last acting [4,2,1]
pg 4.30 is stuck unclean for 165142.078595, current state active+remapped+backfill_toofull, last acting [4,0,2]
pg 4.37 is stuck undersized for 143439.049717, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,0]
pg 4.38 is stuck unclean for 165142.068151, current state active+remapped+backfill_toofull, last acting [1,5,2]
pg 4.3c is stuck unclean for 165141.765477, current state active+remapped+backfill_toofull, last acting [3,5,0]
pg 4.45 is stuck undersized for 143533.778644, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,5]
pg 4.46 is stuck undersized for 143532.613204, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,0]
pg 4.49 is stuck unclean for 165140.114766, current state active+remapped+backfill_toofull, last acting [0,5,2]
pg 4.4a is stuck unclean for 165230.561560, current state active+remapped+backfill_toofull, last acting [2,0,5]
pg 4.4b is stuck undersized for 143533.461485, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,0]
pg 4.4d is stuck undersized for 143440.472946, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,5]
pg 4.52 is stuck unclean for 165368.423967, current state active+remapped+backfill_toofull, last acting [2,1,5]
pg 4.54 is stuck undersized for 143528.930331, current state active+undersized+degraded+remapped+backfill_toofull, last acting [4,1]
pg 4.59 is stuck undersized for 143439.312445, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,0]
pg 4.5c is stuck undersized for 143439.310437, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,1]
pg 4.63 is stuck unclean for 143534.929627, current state active+remapped+backfill_toofull, last acting [5,7,0]
pg 4.c6 is stuck undersized for 156563.117385, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,3]
pg 4.c7 is stuck undersized for 156565.141095, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,1]
pg 4.c9 is stuck undersized for 143439.295249, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,4]
pg 4.ca is stuck unclean for 143536.314308, current state active+remapped+backfill_toofull, last acting [3,1,5]
pg 4.cd is stuck unclean for 165141.970000, current state active+remapped+backfill_toofull, last acting [5,1,3]
pg 4.cf is stuck unclean for 167417.618461, current state active+remapped+backfill_toofull, last acting [2,1,5]
pg 4.d2 is stuck unclean for 165142.068186, current state active+remapped+backfill_toofull, last acting [2,1,5]
pg 4.d3 is stuck unclean for 143536.301519, current state active+remapped+backfill_toofull, last acting [1,3,5]
pg 4.d5 is stuck undersized for 156563.132649, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,2]
pg 4.d7 is stuck undersized for 143532.614019, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,1]
pg 4.db is stuck undersized for 156563.123609, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,3]
pg 4.de is stuck undersized for 143532.617338, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,1]
pg 4.e3 is stuck undersized for 143532.538071, current state active+undersized+degraded+remapped+backfill_toofull, last acting [0,2]
pg 4.e9 is stuck undersized for 143532.558745, current state active+undersized+degraded+remapped+backfill_toofull, last acting [0,3]
pg 4.ea is stuck undersized for 143439.088624, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,3]
pg 4.eb is stuck undersized for 143533.260848, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,1]
pg 4.ee is stuck unclean for 165141.970248, current state active+remapped+backfill_toofull, last acting [1,3,4]
pg 4.f1 is stuck unclean for 157299.925877, current state active+remapped+backfill_toofull, last acting [3,1,5]
pg 4.f5 is stuck undersized for 143534.921381, current state active+undersized+degraded+remapped+backfill_toofull, last acting [0,5]
pg 4.f6 is stuck unclean for 165141.765656, current state active+remapped+backfill_toofull, last acting [0,3,4]
PG_DEGRADED_FULL Degraded data redundancy (low space): 82 pgs backfill_toofull
pg 4.7 is active+remapped+backfill_toofull, acting [2,0,5]
pg 4.8 is active+undersized+degraded+remapped+backfill_toofull, acting [4,3]
pg 4.a is active+remapped+backfill_toofull, acting [5,1,2]
pg 4.e is active+remapped+backfill_toofull, acting [2,1,4]
pg 4.f is active+undersized+degraded+remapped+backfill_toofull, acting [0,2]
pg 4.10 is active+undersized+degraded+remapped+backfill_toofull, acting [3,1]
pg 4.13 is active+undersized+degraded+remapped+backfill_toofull, acting [0,4]
pg 4.19 is active+undersized+degraded+remapped+backfill_toofull, acting [1,2]
pg 4.1a is active+undersized+degraded+remapped+backfill_toofull, acting [3,0]
pg 4.1b is active+remapped+backfill_toofull, acting [3,1,4]
pg 4.1f is active+undersized+degraded+remapped+backfill_toofull, acting [2,1]
pg 4.21 is active+remapped+backfill_toofull, acting [5,1,3]
pg 4.24 is active+remapped+backfill_toofull, acting [1,2,5]
pg 4.28 is active+undersized+degraded+remapped+backfill_toofull, acting [3,0]
pg 4.29 is active+undersized+degraded+remapped+backfill_toofull, acting [3,1]
pg 4.2b is active+remapped+backfill_toofull, acting [4,2,1]
pg 4.30 is active+remapped+backfill_toofull, acting [4,0,2]
pg 4.37 is active+undersized+degraded+remapped+backfill_toofull, acting [3,0]
pg 4.38 is active+remapped+backfill_toofull, acting [1,5,2]
pg 4.3c is active+remapped+backfill_toofull, acting [3,5,0]
pg 4.45 is active+undersized+degraded+remapped+backfill_toofull, acting [3,5]
pg 4.46 is active+undersized+degraded+remapped+backfill_toofull, acting [3,0]
pg 4.49 is active+remapped+backfill_toofull, acting [0,5,2]
pg 4.4a is active+remapped+backfill_toofull, acting [2,0,5]
pg 4.4b is active+undersized+degraded+remapped+backfill_toofull, acting [2,0]
pg 4.4d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5]
pg 4.52 is active+remapped+backfill_toofull, acting [2,1,5]
pg 4.54 is active+undersized+degraded+remapped+backfill_toofull, acting [4,1]
pg 4.59 is active+undersized+degraded+remapped+backfill_toofull, acting [2,0]
pg 4.5c is active+undersized+degraded+remapped+backfill_toofull, acting [2,1]
pg 4.63 is active+remapped+backfill_toofull, acting [5,7,0]
pg 4.c6 is active+undersized+degraded+remapped+backfill_toofull, acting [1,3]
pg 4.c7 is active+undersized+degraded+remapped+backfill_toofull, acting [3,1]
pg 4.c9 is active+undersized+degraded+remapped+backfill_toofull, acting [2,4]
pg 4.ca is active+remapped+backfill_toofull, acting [3,1,5]
pg 4.cd is active+remapped+backfill_toofull, acting [5,1,3]
pg 4.cf is active+remapped+backfill_toofull, acting [2,1,5]
pg 4.d2 is active+remapped+backfill_toofull, acting [2,1,5]
pg 4.d3 is active+remapped+backfill_toofull, acting [1,3,5]
pg 4.d5 is active+undersized+degraded+remapped+backfill_toofull, acting [1,2]
pg 4.d7 is active+undersized+degraded+remapped+backfill_toofull, acting [3,1]
pg 4.db is active+undersized+degraded+remapped+backfill_toofull, acting [1,3]
pg 4.de is active+undersized+degraded+remapped+backfill_toofull, acting [3,1]
pg 4.e3 is active+undersized+degraded+remapped+backfill_toofull, acting [0,2]
pg 4.e9 is active+undersized+degraded+remapped+backfill_toofull, acting [0,3]
pg 4.ea is active+undersized+degraded+remapped+backfill_toofull, acting [1,3]
pg 4.eb is active+undersized+degraded+remapped+backfill_toofull, acting [2,1]
pg 4.ee is active+remapped+backfill_toofull, acting [1,3,4]
pg 4.f1 is active+remapped+backfill_toofull, acting [3,1,5]
pg 4.f5 is active+undersized+degraded+remapped+backfill_toofull, acting [0,5]
pg 4.f6 is active+remapped+backfill_toofull, acting [0,3,4]

 

[tschanness]

Degraded data redundancy (low space): 82 pgs backfill_toofull

Mehr/größere OSDs würde ich sagen.

 

[9acca9]

Degraded data redundancy (low space): 82 pgs backfill_toofull

Mehr/größere OSDs würde ich sagen.

Ok, thanks but there is a way to fix this?

ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 hdd 0.90439 1.00000 926G 501G 424G 54.13 1.03 67
1 hdd 0.90439 1.00000 926G 597G 328G 64.52 1.22 79
6 hdd 1.81360 1.00000 1857G 508G 1348G 27.40 0.52 68
8 hdd 0.90439 1.00000 926G 269G 656G 29.13 0.55 36
2 hdd 0.90439 1.00000 926G 629G 297G 67.93 1.29 84
3 hdd 0.90439 1.00000 926G 608G 317G 65.72 1.24 81
7 hdd 1.81360 1.00000 1857G 618G 1238G 33.32 0.63 83
4 hdd 0.90439 1.00000 926G 825G 100G 89.12 1.69 110
5 hdd 0.90439 1.00000 926G 823G 102G 88.96 1.69 109
TOTAL 10196G 5383G 4813G 52.79

or just have to put more disk??

Thanks!

 

[tschanness]

How are the OSDs spread over servers?

 

[9acca9]

How are the OSDs spread over servers?

Hi, thanks

ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 9.95792 root default
-7 0 host fiorito
-9 4.52676 host hijos
0 hdd 0.90439 osd.0 up 1.00000 1.00000
1 hdd 0.90439 osd.1 up 1.00000 1.00000
6 hdd 1.81360 osd.6 up 1.00000 1.00000
8 hdd 0.90439 osd.8 up 1.00000 1.00000
-3 3.62238 host n1
2 hdd 0.90439 osd.2 up 1.00000 1.00000
3 hdd 0.90439 osd.3 up 1.00000 1.00000
7 hdd 1.81360 osd.7 up 1.00000 1.00000
-5 1.80878 host n2
4 hdd 0.90439 osd.4 up 1.00000 1.00000
5 hdd 0.90439 osd.5 up 1.00000 1.00000

 

[tschanness]

On the Host n2 the OSDs are at 90%:
4 hdd 0.90439 1.00000 926G 825G 100G 89.12 1.69 110
5 hdd 0.90439 1.00000 926G 823G 102G 88.96 1.69 109

Ceph can't balance anymore. Add an OSD to node n2.