remove non-existant node

[Backshift1516]

Hello guys,

I did a little oopsie. I had a Cluster of 3 nodes that I decided to dissolve someday.
The Problem is, that I didn't properly remove the nodes from the cluster before removing them physically.
I watched some Youtube Tutorials and managed to remove the cluster with the config files and all, but I couldn't delete the second of my 3 nodes for some reason.

Some time has passed and it hasn't caused any Problems, but I thought I should remove it at some point.

Since there is no Cluster I also cannot remove the node

root@pve1:~# pvecm nodes
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
root@pve1:~# pvecm delnode pve2
Node/IP: pve2 is not a known host of the cluster.
root@pve1:~#

Any Ideas on how to solve this?

Appreciate any help

 

[sw-omit]

Anything in either of these two folders?:
/etc/corosync
/var/lib/corosync
And is the node also removed from /etc/pve/nodes/pve2 ? (Do NOT empty out the /etc/pve/nodes folder, you'll break your main node too, but you should be able to (re)move the pve2 subfolder.)

Also take a look at the below in the section "Separate a Node Without Reinstalling" in there:
https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

 

[Backshift1516]

Anything in either of these two folders?:
/etc/corosync
/var/lib/corosync
And is the node also removed from /etc/pve/nodes/pve2 ? (Do NOT empty out the /etc/pve/nodes folder, you'll break your main node too, but you should be able to (re)move the pve2 subfolder.)

Also take a look at the below in the section "Separate a Node Without Reinstalling" in there:
https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

The mentioned folders contain some files:

root@pve1:~# ls /etc/corosync
authkey  uidgid.d
root@pve1:~# ls /var/lib/corosync
fdata  fdata-2023-07-15T15:07:00-965  ringid_1
root@pve1:~# ls /etc/pve/nodes
pve1  pve2  pve3
root@pve1:~#

interestingly all three still remain here

 

[sw-omit]

Ok, checking it with one of my own, non-clustered, servers, the /etc/corosync/uidgid.d folder exists, but is empty (and no authkey file), the /var/lib/corosync folder is empty, and the /etc/pve/nodes only contains the folder of that node.
So I would suggest to MOVE the following things to someplace else, check if everything still works, reboot, then check again (and then optionally remove those files/folders fully).

• authkey-file
• anything INSIDE of the uidgid.d folder (if anything)
• the contents of the /var/lib/corosync folder (again, not the folder itself)
• the pve2 and pve3 folders of the /etc/pve/nodes

 

[tcabernoch]

You got some good suggestions above for fixing this in normal circumstances.
Follow all of that advice, but if it doesn't work ...

Check your cluster HA config. See anything weird there? Ghost machines maybe? Kill what ya can.

Then, On each node in the cluster, one after the other ...
systemctl restart pve-cluster
systemctl restart corosync

 

[Backshift1516]

Ok, checking it with one of my own, non-clustered, servers, the /etc/corosync/uidgid.d folder exists, but is empty (and no authkey file), the /var/lib/corosync folder is empty, and the /etc/pve/nodes only contains the folder of that node.
So I would suggest to MOVE the following things to someplace else, check if everything still works, reboot, then check again (and then optionally remove those files/folders fully).

• authkey-file
• anything INSIDE of the uidgid.d folder (if anything)
• the contents of the /var/lib/corosync folder (again, not the folder itself)
• the pve2 and pve3 folders of the /etc/pve/nodes

It seems like this worked. I guess I will keep the copy of the files for now... just in case smth goes wrong.

Thank you for your help!