deleted cluster node still shows up in my gui

[smcbutler]

hi, i deleted a node from my cluster without any problems but the deleted node still shows up in the proxmox gui.

here's my gui screen show
http://cl.ly/KSQz

and i've included below what pvecm is telling me i have in my cluster. as you can see they're different. does anyone know how i can clean up my gui to only show real nodes in the cluster?

Thx


root@ovz1:~# pvecm nodes
Node Sts Inc Joined Name
1 M 180 2012-10-27 01:05:11 ovz1
root@ovz1:~# pvecm status
Version: 6.2.0
Config Version: 3
Cluster Name: evals
Cluster Id: 3279
Cluster Member: Yes
Cluster Generation: 180
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: ovz1
Node ID: 1
Multicast addresses: 239.192.12.219
Node addresses: 10.8.4.2

 

[dietmar]

hi, i deleted a node from my cluster without any problems but the deleted node still shows up in the proxmox gui.

You need to remove the whole directory /etc/pve/nodes/<nodename>

 

[smcbutler]

that worked - thanks!

 

[w3ph]

You need to remove the whole directory /etc/pve/nodes/<nodename>

Has this changed in 2.2? I deleted a node that will be retired and deleted the directory from /etc/pve/nodes/<nodename> and the node is still there in the web interface, marked in red. The only mention I see of the node is in /etc/pve/.members - it's still listed there. Also in /var/lib/pve-cluster/config.db

I'm missing a step but not sure what it is.

/Paul

 

[dietmar]

I'm missing a step but not sure what it is.

How did you remove the node exactly?

 

[w3ph]

How did you remove the node exactly?

On the 'master' node:

pvecm delnode prox2003
rm -rf /etc/pve/nodes/prox2003

Possible clue: this resulted in 'pvecm status' reporting one more expected votes than there were nodes, so I changed expected votes to match the new number of nodes.

Also, pvecm nodes still shows the just-deleted node (prox2003) and one I deleted a while ago that does not appear in the web gui:

Node Sts Inc Joined Name
1 M 316 2012-06-30 03:01:36 prox2001
2 X 320 prox2002
3 M 2076 2013-02-10 06:08:30 prox2014
4 X 2036 prox2003
5 M 1480 2012-09-12 14:47:14 prox2010
6 M 1480 2012-09-12 14:47:14 prox2009

version: pve-manager/2.2/3089a616

/Paul

 

[dietmar]

Does it help if you restart pvestatd

# service pvestatd restart

 

[w3ph]

No difference. I just ran service pvestatd restart on node 1 again, no change.

/Paul

 

[w3ph]

What about /etc/pve/.members - that file still lists the deleted node - there's no IP number, it's still counted as a node in the node count - it looks like this:

"prox2003": { "id": 4, "online": 0},

That file is write-protected, even from root - is there a safe way to remove this line, and would that help?

 

[dietmar]

That file is write-protected, even from root - is there a safe way to remove this line, and would that help?

That is a magic file, you cant remove it. But you can restart the file system

# service pve-cluster restart

 

[w3ph]

That is a magic file, you cant remove it. But you can restart the file system

# service pve-cluster restart

No change - the /etc/pve/.members file was rebuilt, but still shows the removed node (which really is gone).

If I try to remove the node again pvecm tells me that it's already gone:

root@prox2001:~# pvecm delnode prox2003
node prox2003 does not exist in /etc/pve/cluster.conf

grepping for the name of the node, I see that the removed node is still in /var/lib/pve-cluster/config.db - is there a way to rebuild that file?

 

[w3ph]

Hmmm - I "fixed" the problem by accident. I added a new node (new server, previously unused hostname) to the cluster. That regenerated whatever needed to be regenerated, and now the ghost server does not appear in pvecm nodes or in the GUI.

Problem solved until next time I delete a node ;-)

 

[Matteo Giacomelli]

I had that problem too. I noticed that the deleted node was defined also in /etc/corosync/corosync.conf file
So following this helpful guide (https://pve.proxmox.com/wiki/Editing_corosync.conf) I was able to deleted from GUI the deleted node.

 

[Oliver Krüger]

From two out of three cluster nodes the forth (dead) node has disappeared from the webui. Tried all the tricks above - no luck.
On node3 the 4th node's name is still mentioned in /etc/pve/.members and /etc/pve/ha/manager_status.

Is it safe to reinstall node4 (with the old name and IP addresses) and re-add it to the cluster? Or should I purge the forth's name from the history books first? If the latter, how?

 

[Stoiko Ivanov]

From two out of three cluster nodes the forth (dead) node has disappeared from the webui. Tried all the tricks above - no luck.
On node3 the 4th node's name is still mentioned in /etc/pve/.members and /etc/pve/ha/manager_status.

That sounds like node3 is not in the cluster's quorum:
* check the output of `pvecm status` (on all nodes)
* check for potential problems in the logs - `journalctl -r` gives you the logs in reverse order - look for messages from corosync and pmxcfs

make sure all nodes of your cluster are inside the quorum before adding/removing other nodes.

The addition/removal of nodes to the clusters is quite well explained in the reference documentation - https://pve.proxmox.com/pve-docs/chapter-pvecm.html

hope this helps!

 

[Oliver Krüger]

All three remaining nodes show the same "pvecm status" output (except for the nodeid).
Logs don't show anything noteworthy.

Anything else I should/can check?

Probably the 4th node afterfglow is gone after a reboot. Unfortunately rebooting the machine is not an option in my case.

Quorum information
------------------
Date: Thu Jun 20 09:05:18 2019
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 1/432
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.0.0.1
0x00000002 1 10.0.0.2
0x00000003 1 10.0.0.3 (local)

 

[Stoiko Ivanov]

Hmm - if all nodes show the same - the 4th node is not in the cluster anymore.
If you still see the fourth node in '/etc/pve/.members' on one node you should try to restart 'pve-cluster.service' and maybe 'corosync.service' on the node where it's still there. Also make sure that the 4th node is not connected to the cluster-network anymore

hope this helps!

btw. I just noticed - this thread is quite old - please open a new one in such cases instead of replying to a thread originally from 6 years ago - Thanks.

 

[Oliver Krüger]

That helped. Thank you very much.

I already restarted pve-cluster and corosync services before. The difference this time was, I powered off the forth node machine and therefore made it unavailable while restarting the services. (There was a fresh installation of Proxmox running on the forth node (with no configuration whatsoever). Maybe that was blocking the afterglow in the other webui/.members-file.)

No need to start a fresh thread. Case closed. ;-)

 

[tisc0]

My 2 pennies thoughts:
I think it's better to have an answer on an old topic actually. We don't lose the history and all the things tried, the good and bad ideas, the one working, the ones not working. It's creating coherence, and a more complete reference on a problem. New ones should be redirected by moderators on old one already talking about the same subject. Otherwise, we reinvent the wheel everytime.

Now, going to surf again to maybe find a solution at that problem. Have a good day

 

[randre.wright]

You need to remove the whole directory /etc/pve/nodes/<nodename>

Thank you. This worked for me too.