[SOLVED] remove damaged ceph node

[lordhanuman]

Hi all, I have a 3-node cluster, one died badly. I re-added a third one, I removed the node from the cluster, but I can't figure out how to remove it from ceph, so it stays inside the cluster and I can't remove it. I've looked in the documentation but I don't understand how to remove it.
the failed node had a vps inside. it makes me go to the node and I have the disk inside ceph but I don't know how to remove it
Thank you

 

[dcsapak]

you probably still have the config file left in '/etc/pve/nodes/NODENAME/qemu-server' you could simply move that config into the directory of a working node, then it should appear there and if you remove it, it should remove the ceph images as well

 

[lordhanuman]

I managed to remove the vps by moving the file as you suggested. Now to remove the node from the list and ceph disks how does it work? I already did the uncluster procedure

 

[dcsapak]

like i wrote, if you move the config file to e.g. '/etc/pve/nodes/WORKINGNODE/qemu-server' the vm should appear on 'WORKINGNODE' (replace that with a real nodename) and then you should be able to remove either the disks from there or the vm as a whole

 

[lordhanuman]

I removed the VPS and the disk and I was able to remove the node from the list by deleting the directory /etc/pve/nodes/brokennode

Now I would like to figure out how to remove the ceph references to the broken node
Thank you

 

[dcsapak]

ok that should be doable by following the ceph docs: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-monitors

edit: and the old nodes should be fixable by removing the corresponding node folder in /etc/pve/nodes/ (make a backup if you're not sure if you still need it)

 

[lordhanuman]

ok that should be doable by following the ceph docs: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-monitors

edit: and the old nodes should be fixable by removing the corresponding node folder in /etc/pve/nodes/ (make a backup if you're not sure if you still need it)

Ok

ok i managed to remove broken node from ceph with command
ceph mon remove {mon-id}
Thanks