Proxmox VE with Ceph having PG warning after removing 3x OSD and rebuilt 3x OSD

[Anson Sea]

Hi All,

I simulated removing 1x HDD from each of my 3x Host of Proxmox VE which configured as OSD for CEPH.
I then plug them back in and destroyed the OSD > Wipe the Disk > recreate OSD.
From then I face this warning PG 2.10 unknown issue, I wasnt able to clear.
I have tried remove the .mgr > delete active manager under monitor > recreate manager > ensure .mgr recreated, but there's no luck.
I have confirmed all existing OSD are serving properly but I couldnt get this warning away and it is affected the system.

 

[robertlukan]

Have you waited for Ceph to recover after removing each OSD ?

 

[Anson Sea]

Hi Robert, thank you for replying. I have 4 OSD on each host, hence I was simulating removing 1 OSD from each host at the same time. I have confirmed that after I slot in the HDD and rebuilt the OSD, the peering shows completed.

Have you waited for Ceph to recover after removing each OSD ?

 

[gurubert]

You have lost data.

 

[robertlukan]

Yes, most probably you have lost data. As you are having 3 copies of every PG, it can happen that PG was located on those three disks, so when you have removed them at the same time it became unavailable. As you have wiped OSDs, I dont think there is much to do now, hopefully this was test setup or you have backups.

You can shutdown one host, or remove one disk at the same time and you will be fine. If you shutdown 2 hosts or remove 2 disks from different hosts, your data will be read-only.

 

[UdoB]

hence I was simulating removing 1 OSD from each host at the same time.

That's just not how it works.

"Failure domain" is host. You could have shutdown one host with all 4 OSDs going down. That would have worked fine.

All data is distributed between those three nodes. You do not know onto which OSD. (Well, you could learn that from some debugging, but during normal operation that is irrelevant und unknown.) When you remove one random OSD from each host, the chance that you killed more than one copy of a specific block is high..

 

[alexskysilk]

I simulated removing 1x HDD from each of my 3x Host of Proxmox VE which configured as OSD for CEPH.

You will lose pg coherency on some pgs, and lose entire pgs for others.

From then I face this warning PG 2.10 unknown issue, I wasnt able to clear.

if you no longer have at LEAST one OSD that contained a shard for that pg, you will need to destroy that pg manually.

 

[Anson Sea]

thank you everyone for your response, may I check if I remove 1 OSD from 2 host, will it kill the ceph?

 

[Anson Sea]

You will lose pg coherency on some pgs, and lose entire pgs for others.


if you no longer have at LEAST one OSD that contained a shard for that pg, you will need to destroy that pg manually.

Thank you Alex, may I check how to destroy the pg manually? I cant seems to find a way to do it.

 

[alexskysilk]

I cant seems to find a way to do it.

Have you tried looking at the documentation? https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/