Not able to remove cluster node

[yousuf101]

I deployed a third node and tried to add it to the cluster, but the process failed. When I run pvecm nodes, only the two existing nodes are shown; the new node does not appear. However, it is visible in the GUI interface.

How can I remove the dead server from the list if it's not showing in the shell command? Is there a way to remove the dead node from the list so I can add the new rebuilt node?

 

[gfngfn256]

Have you tried rebooting the 2-node cluster?
You could also try:

systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd

 

[justinclift]

... tried to add it to the cluster, but the process failed.

Do you remember what the error message was?

For the node showing up in the webUI, that probably means it now has an entry defined under /etc/pve/nodes/NODE_NAME.

Saying that as "a phantom node showing up in the webUI" also happens when you delete a node from an existing cluster.

It'll be gone from the cluster definition, but still show in the web interface until you manually nuke it's directory structure under /etc/pve/nodes/NODE_NAME.

To stop it from showing up, I'd first confirm if there is an entry for that phantom node in the above directory, and if so then nuke it (rm -rf style).

With that gone, you should be able to reload your browser tab (Control-F5 on Firefox, not sure about Chrome) and the entry should be gone from the webUI.

The question after that one is "how to get the cluster join operation to work", which the error message would be helpful for.

 

[yousuf101]

this error when click on the failed host

and when click on cluster tap

 

[justinclift]

Interesting. That sounds like a (name) resolver problem of some sort. Unfortunately my skills aren't strong in that area.

For the basic bits though, if you try pinging that name (ie "ping IEDUBPVE103") from both of the other hosts, does it work?

 

[justinclift]

Hmmm, doing a search here on the Proxmox forums for that error message "no address associated with hostname" does show some potentially useful threads.

I'd probably look into those first (after fixing the phantom entry in the web interface), as my skills with fixing name resolution problems are likely not as good as some of the other people in those threads.

 

[DJFriar]

For the node showing up in the webUI, that probably means it now has an entry defined under /etc/pve/nodes/NODE_NAME.

It'll be gone from the cluster definition, but still show in the web interface until you manually nuke it's directory structure under /etc/pve/nodes/NODE_NAME.

To stop it from showing up, I'd first confirm if there is an entry for that phantom node in the above directory, and if so then nuke it (rm -rf style).

I am unable to nuke the folder located here. In my case it is /etc/pve/nodes/pve2 and it has four empty directories within it; but nothing I do will let me remove the folder. I do see that the Write permissions are not present, even as root, so I assume that is the issue, but chmod 777 isn't working either.

 

[Verulam]

For stubborn hosts I find this works, which removes and resets them:

systemctl stop pve-cluster
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -r /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster

PVE Wiki

 

[justinclift]

I am unable to nuke the folder located here.

At a guess, it sounds like you're trying to do it when your cluster doesn't have quorum.

Try running pvecm expected 1 first, to force the node you're on to have quorum. Then try nuking the folder again.

 

[justinclift]

or stubborn hosts I find this works

Ugh, don't do that unless you've tried all the other options first. That seems like it'd permanently wipe your entire cluster configuration, which is not the same as removing the details of a single node.

 

[vlalwani]

for anyone still trying this, removing the node from /etc/pve/corosync.conf and restarting pve-cluster and corosync worked for me.