[SOLVED] Ceph Health - backfillfull / OSDs marked as out and down

ssaman · May 9, 2019

Hello proxmox community,

today we noticed our heath error with the message:

Code:

HEALTH_ERR 1 backfillfull osd(s); 1 nearfull osd(s); 1 pool(s) backfillfull; Degraded data redundancy: 99961/8029671 objects degraded (1.245%), 19 pgs degraded, 19 pgs undersized; Degraded data redundancy (low space): 19 pgs backfill_toofull

For more Detail:

Code:

HEALTH_ERR 1 backfillfull osd(s); 1 nearfull osd(s); 1 pool(s) backfillfull; Degraded data redundancy: 99961/8029671 objects degraded (1.245%), 19 pgs degraded, 19 pgs undersized; Degraded data redundancy (low space): 19 pgs backfill_toofull
OSD_BACKFILLFULL 1 backfillfull osd(s)
    osd.10 is backfill full
OSD_NEARFULL 1 nearfull osd(s)
    osd.13 is near full
POOL_BACKFILLFULL 1 pool(s) backfillfull
    pool 'hdd_mainpool' is backfillfull
PG_DEGRADED Degraded data redundancy: 99961/8029671 objects degraded (1.245%), 19 pgs degraded, 19 pgs undersized
    pg 10.0 is stuck undersized for 304111.529099, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,6]
    pg 10.3 is stuck undersized for 1702843.879105, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,8]
    pg 10.4f is stuck undersized for 304111.866785, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.59 is stuck undersized for 304111.866899, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.6b is stuck undersized for 1702843.872904, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,7]
    pg 10.a3 is stuck undersized for 1702843.822618, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,9]
    pg 10.a7 is stuck undersized for 304111.865494, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.ba is stuck undersized for 304111.529376, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,6]
    pg 10.c7 is stuck undersized for 304111.866182, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.d4 is stuck undersized for 1702843.875008, current state active+undersized+degraded+remapped+backfill_toofull, last acting [2,7]
    pg 10.106 is stuck undersized for 1702843.815712, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,7]
    pg 10.109 is stuck undersized for 1702843.856827, current state active+undersized+degraded+remapped+backfill_toofull, last acting [8,3]
    pg 10.110 is stuck undersized for 1702843.814649, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,9]
    pg 10.12b is stuck undersized for 1702843.808098, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,9]
    pg 10.1a4 is stuck undersized for 304111.867262, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.1c2 is stuck undersized for 1702843.812091, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,8]
    pg 10.1d9 is stuck undersized for 304111.867532, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,6]
    pg 10.1e1 is stuck undersized for 1702843.817784, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,8]
    pg 10.1e8 is stuck undersized for 1702843.813599, current state active+undersized+degraded+remapped+backfill_toofull, last acting [3,9]
PG_DEGRADED_FULL Degraded data redundancy (low space): 19 pgs backfill_toofull
    pg 10.0 is active+undersized+degraded+remapped+backfill_toofull, acting [1,6]
    pg 10.3 is active+undersized+degraded+remapped+backfill_toofull, acting [2,8]
    pg 10.4f is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.59 is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.6b is active+undersized+degraded+remapped+backfill_toofull, acting [2,7]
    pg 10.a3 is active+undersized+degraded+remapped+backfill_toofull, acting [1,9]
    pg 10.a7 is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.ba is active+undersized+degraded+remapped+backfill_toofull, acting [1,6]
    pg 10.c7 is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.d4 is active+undersized+degraded+remapped+backfill_toofull, acting [2,7]
    pg 10.106 is active+undersized+degraded+remapped+backfill_toofull, acting [3,7]
    pg 10.109 is active+undersized+degraded+remapped+backfill_toofull, acting [8,3]
    pg 10.110 is active+undersized+degraded+remapped+backfill_toofull, acting [3,9]
    pg 10.12b is active+undersized+degraded+remapped+backfill_toofull, acting [3,9]
    pg 10.1a4 is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.1c2 is active+undersized+degraded+remapped+backfill_toofull, acting [1,8]
    pg 10.1d9 is active+undersized+degraded+remapped+backfill_toofull, acting [3,6]
    pg 10.1e1 is active+undersized+degraded+remapped+backfill_toofull, acting [3,8]
    pg 10.1e8 is active+undersized+degraded+remapped+backfill_toofull, acting [3,9]

Also, there are 2 OSDs down / out from nothing.

Code:

ceph osd tree
ID  CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
 11   hdd  5.45740         osd.11   down        0 1.00000
 12   hdd  5.45789         osd.12   down        0 1.00000

We set the OSDs back to "in" and tried to start the OSDs but we had no success (no response)

Code:

sudo systemctl start ceph-osd@12
sudo systemctl start ceph-osd@11

tim · May 9, 2019

What's your question? As you can read from the health status output there isn't enough space left in your cluster.
http://docs.ceph.com/docs/luminous/rados/troubleshooting/troubleshooting-osd/?#no-free-drive-space

Check your logs why those osds are down, perhaps they were near full as well?

Logs: /var/log/ceph/<osd-which-failed>

ssaman · May 9, 2019

There are some errors that I can't interpret or know how to fix like:

Code:

/var/log/ceph/ceph-osd.11.log

Code:

     0> 2019-05-09 13:03:43.860878 7fb79ab8e700 -1 /mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7fb79ab8e700 time 2019-05-09 13:03:43.859862
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: 8877: FAILED assert(r == 0)

Code:

    -3> 2019-05-09 13:03:42.882799 7fb79938b700  2 rocksdb: [/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/rocksdb/db/db_impl_compaction_flush.cc:1275] Waiting after background compaction error: Corruption: block checksum mismatch, Accumulated background error counts: 1

tim · May 9, 2019

Can you please post more than just these 2 lines?
Before digging into this, was your cluster near full before those osds got down?

ssaman · May 9, 2019

tim said:
Before digging into this, was your cluster near full before those osds got down?

No, the cluster wasn't near full.

Current Usage:

Maybe the problem is that we set our replica count to 3.
And we have 4 OSDs each node.

tim said:
Can you please post more than just these 2 lines?

I have uploaded the log file. I hope it helps.

ssaman · May 14, 2019

We fixed the problem by removing the disks and reuse the same disk again. We don't know why Ceph throw out the OSDs even though the disks are still good.
Currently, we struggle with an LSI MegaRAID Controller.
We put back the same disk in the same slot from the RAID-Controller (RAID0 / single non RAID disk) but the controller doesn't pass through the disk to Proxmox.

tim · May 14, 2019

We strongly advise against any RAID-Controller with Ceph and I'm very sorry that you are facing this issues, but that's exactly why we always advise to not use raid controllers with ceph or zfs.

ssaman · May 14, 2019

Thank you, Tim for the information.

If anyone wanted to know how we fixed it right now.

Like already mentioned before we have an LSI MegaRAID Controller MR9260-i4. This Controller isn't able to set a disk to JBOD-Mode.
The workaround is to create a RAID0 with a single disk.

We noticed that the RAID0 associated to the disk we missed was gone.
We don't know how but it happened. So the disk wasn't in any RAID (unconfigured(good)).

So we created a new RAID0 and added the disk to the raid.

There we were able to see the disk again on the OS.

Our Ceph status changed from error to warning. We just have to wait that Ceph is healing "misplaced objects".

Alwin · May 16, 2019

ssaman said:
So we created a new RAID0 and added the disk to the raid.

The RAID leves or JBOD are really not made for the Ceph use case. I have experienced this myself very badly. Please see our precondition section in the docs for more information on hardware.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_precondition

Please consider, if possible, to by HBAs. It makes Ceph a lot easier to handle.

Search

Search

[SOLVED] Ceph Health - backfillfull / OSDs marked as out and down

ssaman

Active Member

tim

Proxmox Staff Member

ssaman

Active Member

tim

Proxmox Staff Member

ssaman

Active Member

Attachments

ssaman

Active Member

tim

Proxmox Staff Member

ssaman

Active Member

Alwin

Proxmox Retired Staff