[SOLVED] Remove node from cluster

[alterman1994]

Hello Guys,

I mistakenly added new node to cluster using public network (instead of storage one). I'm checking https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

to remove the node. I powered off the machine (pve7) to make sure it will never power on again (going to reconfigure it from scratch)



However, when I execute pvecm nodes, I don't see this node listed

root@pve1:~# pvecm nodes


Membership information
----------------------
    Nodeid      Votes Name
         1          1 pve1 (local)
         2          1 pve2
         3          1 pve3
         4          1 pve4
         5          1 pve5
         6          1 pve6

How can I properly remove it? Could someone please assist? The cluster is in production, don't want to screw it up.

Proxmox version: 6.4-13

Thanks.

 

[mira]

Yes, pvecm nodes only lists the active ones.
In the GUI you see the name of the node you want to delete. Use that for the command pvecm delnode as described in the docs/wiki.

 

[alterman1994]

@mira Hello, thank you for your feedback.

I run it and that's what I got:

root@pve1:~# pvecm delnode pve7
Could not kill node (error = CS_ERR_NOT_EXIST)
Killing node 7
command 'corosync-cfgtool -k 7' failed: exit code 1

Seems like it did the trick, I don't see pve7 anymore in gui, /etc/corosync/corosync.conf and /etc/pve/corosync.conf

However, do I need to worry about the output? Anything I need to reload after this operation? Sorry if questions look stupid to you.

Thanks in advance.

 

[alterman1994]

Nevermind, I've reinstalled it from scratch, cleaned old known_hosts and added new node properly.

Thanks for your great support!

 

[proxwolfe]

I, too, am trying to remove a node from my cluster (to set it up anew and rejoin it).

I managed to delete it using pvecm but it still showed up in the GUI.

So I removed the respective folder under /etc/pve/nodes and now it is gone.

But doing this, I discovered that I have another folder /etc/pve/nodes/pve which does not correspond to a node shown in the GUI or still in existence. My cluster has evolved over time and I removed all original nodes and replaced them with newer ones. It *may* be that one of the originals nodes was named "pve". Can I simply remove also this folder or might I kill my cluster?

Thanks

 

[Jannoke]

You probably won't kill the clustet, but you can backup it and remove it and test if something goes wrong. You can always put it back. Just don't do it on working hours. Anyways thanks for the tip on removing the /etc/pve/nodes/<node> folder. I got to same point where it deleted the node but did not remove it from GUI.

 

[cryptocurt]

I, too, am trying to remove a node from my cluster (to set it up anew and rejoin it).

I managed to delete it using pvecm but it still showed up in the GUI.

So I removed the respective folder under /etc/pve/nodes and now it is gone.

But doing this, I discovered that I have another folder /etc/pve/nodes/pve which does not correspond to a node shown in the GUI or still in existence. My cluster has evolved over time and I removed all original nodes and replaced them with newer ones. It *may* be that one of the originals nodes was named "pve". Can I simply remove also this folder or might I kill my cluster?

Thanks

How did you remove the folder, I get an error; rm: cannot remove 'local': Is a directory I tried the -f flag but that's not working

 

[kajizta]

How did you remove the folder, I get an error; rm: cannot remove 'local': Is a directory I tried the -f flag but that's not working

use rm -rf /path/to/foo to delete folder

 

[hrz]

You probably won't kill the clustet, but you can backup it and remove it and test if something goes wrong. You can always put it back. Just don't do it on working hours. Anyways thanks for the tip on removing the /etc/pve/nodes/<node> folder. I got to same point where it deleted the node but did not remove it from GUI.

THANK YOU, this worked, you can use the command "rm -rf /etc/pve/nodes/NAME" reboot and the pesky node is finally gone from the dashboard...

 

[ihr]

On Proxmox 7.3 after running

pvecm delnode <cabecicubo>

The node got deleted, and pvecm nodes no longer shows cabecicubo on the list; but, in the UI, the node is still visible, without the green check, but visible.

How can I remove it from the UI?

 

[proxwolfe]

Have you tried removing the folder corresponding to the node under /etc/pve/nodes?

 

[ihr]

Have you tried removing the folder corresponding to the node under /etc/pve/nodes?

No, That directory exist. In that folder there is "old" information about that node but the interesting thing is that in the sub-folder called qemu-server there is still one file 155.conf that contains the configuration of one VM that I no longer use.

The cleanest think to do is to remove all resources that corresponds with that VM and then, remove the node info for that folder.

 

[Jannoke]

I just found that there was still my old node data available on the /etc/pve/nodes path. So just to test it out I just created dummy 1.conf file in the qemu-server subfolder and it immidiately added this node to he list of nodes in web ui. Of course it's with quiestion mark. After removing the 1.conf file, the node will not have any virtual machines, but wil not dissapear from the list until i refresh browser. So yes - qemu-server folder needs to be empty for it to dissapear from nodes tree.

 

[fpdragon]

Hi there,

I think I am currently in a similar problem. The whole mess kind of started when I tried to create a cluster between two different versions of pve nodes. I have learned now and tried to cleanup everything and apgrade both nodes. now they are on same version 7.4-16 .

However I still see the old node after I create a NEW cluster with a new name.
Since a manually removed several configuration files related to cluster and corosync modules before, I guess the problem relies in the /etc/pve/nodes/* directories.

However, maybe I am dumb but I was not able to remove them. I also tried with rm -rf /etc/pve/nodes/nodename but I always get a "Permission denied". There are several subdirs (lxc, openvz priv qemu-server) that I also can't remove the subdirs.

Another problem is, when I execute systemctl stop pve-cluster my remote console connection drops and I am not able remove the directories afterwards. Currently I dont have a physical direct console available so I do have to use the web interface or SSH shell.

Any idea how to get rid of the directories? Thanks.

 

[fpdragon]

Ok, I figured out how to remove the directories. The trick was once again, to remove the whole cluster with

rm -f /etc/pve/cluster.conf /etc/pve/corosync.conf
rm -f /etc/cluster/cluster.conf /etc/corosync/corosync.conf
rm /var/lib/pve-cluster/.pmxcfs.lockfile

After reboot the cluster disappeard and I also was able to remove the node directories.

So it seems that everything was clean now and when creating a new cluster no old node shows up any more.

However that has not solved my main problem that I can't add my second node to the cluster. I guess I will create a new post for that.

 

[Jannoke]

I believe permission denied error came from you not having chorus - You should have at least 3 nodes so that chorus would work. If you have two - as soon as one breaks , the whole cluster will go read-only, because remaining node does not have confirmation if he's "in" or "out". Third node can be some simple NUC or rasperry pi. Does not have to be fullblown machine bearing load.

 

[fpdragon]

I believe permission denied error came from you not having chorus - You should have at least 3 nodes so that chorus would work. If you have two - as soon as one breaks , the whole cluster will go read-only, because remaining node does not have confirmation if he's "in" or "out". Third node can be some simple NUC or rasperry pi. Does not have to be fullblown machine bearing load.

yeah, I do know that nomally 3 nodes are required. I was using 2 nodes before since is still quite handy to migrate vms between the two nodes.

 

[watersheep]

hello im having a isue with my node not booding after a cup failuer i can no longer boot up this node how to i remove the nde from my cluster so i can use my vms after when i get to the gui all i see is this (image 1) i can not boot any vm i get the err code 500 sombody help me

 

[fahadshery]

On Proxmox 7.3 after running

pvecm delnode <cabecicubo>

The node got deleted, and pvecm nodes no longer shows cabecicubo on the list; but, in the UI, the node is still visible, without the green check, but visible.

How can I remove it from the UI?

I have the same issue. I deleted a node but it's still visible in the GUI. Did you solve it or re-install is the only solution which is bonkers!!

 

[mira]

I have the same issue. I deleted a node but it's still visible in the GUI. Did you solve it or re-install is the only solution which is bonkers!!

Check if there are still leftover files in /etc/pve/nodes/<deleted-node>.
As long as there are files still in there, the node will show up in the UI.
Please make a backup of the files before you delete them, just to be safe!