incomplete PGs

[luphi]

Hey guys,

this time I really need your help :-(
first my setup:
3 nodes (pve0,pve1,pve2)
2 pools (both of size 2/1)
all nodes are running latest version 5.0/12.2.0

I just wanted to migrate all OSDs from filestore to bluestore, so I removed all OSDs of node pve2 and recreated them. Unfortunately I haven't noticed another OSD (pve0,osd.19) was down for any reason. Since then I have 5 incomplete PGs.

Any help is really, really apprechiated.
Please let me know, what further information you need to troubleshoot.

Cheers,
luphi

 

[Alwin]

Well, do you have backups? These 5 PGs are very likely lost, as there are no copies around anymore.

To get the statistics for all placement groups stuck in a specified state, execute the following:

ceph pg dump_stuck inactive|unclean|stale|undersized|degraded [--format <format>] [-t|--threshold <seconds>]

Inactive Placement groups cannot process reads or writes because they are waiting for an OSD with the most up-to-date data to come up and in.
Unclean Placement groups contain objects that are not replicated the desired number of times. They should be recovering.
Stale Placement groups are in an unknown state - the OSDs that host them have not reported to the monitor cluster in a while (configured by mon_osd_report_timeout).

Valid formats are plain (default) and json. The threshold defines the minimum number of seconds the placement group is stuck before including it in the returned statistics (default 300 seconds).

Depending on your setup and state, you create a new pool with size 3 / min_size 2 and try to move your VMs there or try to restore them from backup.

 

[luphi]

Hello Alwin,

yes, that's what I expected. :-(
Any idea, how to find out, which VM is affected?

Cheers,
luphi

 

[Alwin]

This script could be of help, but no warranty that it works. With a second grep you might be able to find the object and correlate to its vmid disk.
http://www.sebastien-han.fr/blog/2013/11/19/ceph-rbd-objects-placement/

 

[luphi]

Thank you for the link, will give it a try...