Ceph cluster completely lost

[mgiammarco]

Hello,
I have an 8 hosts proxmox+ceph cluster.
Yesterday one hdd broke. While reconstructing proxmox ha decided to hard reboot some servers.

Now ceph has some incomplete pgs and refuses to do i/o on all pool.
So I cannot start and/or recover any vm.

What can I do?

I am in contact with ceph guys but they told me to ask also here.

Thanks,
Mario

 

[tom]

without very detailed information it would be hard to help.

pls post more details about your problem.

 

[udo]

Hello,
I have an 8 hosts proxmox+ceph cluster.
Yesterday one hdd broke. While reconstructing proxmox ha decided to hard reboot some servers.

Now ceph has some incomplete pgs and refuses to do i/o on all pool.
So I cannot start and/or recover any vm.

What can I do?

I am in contact with ceph guys but they told me to ask also here.

Thanks,
Mario

Hi,
like Tom wrote - not easy to say without more infos...

Look, which OSDs are down with

ceph osd tree

and try to get as much as possible to bring up (depends on your replica/crush one host can fail).

Further info with

ceph health detail
ceph -s

Udo