How to get rid of dead node in a cluster?

[davidand]

I have a hardware failure, one of my nodes died.

I wanted to remove it via CLI:

# pvecm delnode proxmox-4
Killing node 4
Could not kill node (error = CS_ERR_NOT_EXIST)
command 'corosync-cfgtool -k 4' failed: exit code 1

Now, in the web interface, the node is still there, but almost empty -- and I don't know how to get rid of it?



Repeating this command does not help:

# pvecm delnode proxmox-4
Node/IP: proxmox-4 is not a known host of the cluster.

Thanks!

 

[VictorSTS]

Check with pvecm status in the votequorum information section how many votes your cluster is expecting. Maybe the node got deleted but you browser is still caching the webUI with the old node.

 

[davidand]

Hi @VictorSTS - I had three nodes, one died, so removing the third one would surely break the quorum, however, the hardware is gone so I need to remove it anyway pvecm status is below - does it tell anything, in your opinion?

# pvecm status
Cluster information
-------------------
Name:             home-proxmox
Config Version:   26
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sun Feb 25 09:58:58 2024
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          2.1501
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 192.168.1.6 (local)
0x00000003          1 192.168.1.7

 

[VictorSTS]

Looks like pvecm delnode proxmox-4 succeded, as you only have 2 nodes in your cluster now:

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 192.168.1.6 (local)
0x00000003 1 192.168.1.7[/CODE]

 

[esi_y]

Now, in the web interface, the node is still there, but almost empty -- and I don't know how to get rid of it?

rm -rf /etc/pve/nodes/proxmox-4

This needs to be executed on one of the remaining nodes. GUI refresh afterwards.

 

[davidand]

@tempacc346235 - thanks, this helped indeed:

rm -rf /etc/pve/nodes/proxmox-4

 

[VictorSTS]

I don't recall having to remove any directory and I've removed quite a few nodes. Do you know why in this case he had to remove it manually? Maybe related to that old 7.0 version?

 

[davidand]

@VictorSTS I have no idea However, removing the folder (on the same machine where I earlier run 'pvecm delnode') did help to get rid of the entry in web ui.

 

[homelabenthusiast]

in the newer version of i always have to execute rm -rf /etc/pve/nodes/<NODE Name> to remove it in the GUI. Also done this many times. Adding, removing and reusing nodes and readding them. Also did that when i renamed a node in my cluster because it created a new entry

 

[esi_y]

@fiona told me once before it was not automatically removed just in case user wants to copy something out of there (VM configs?). I am not familiar with older versions or what had changed. It should have been now added to the wiki into the manual steps for removing a node to also wipe the directory.

 

[homelabenthusiast]

@fiona told me once before it was not automatically removed just in case user wants to copy something out of there (VM configs?). I am not familiar with older versions or what had changed. It should have been now added to the wiki into the manual steps for removing a node to also wipe the directory.

I just checked the wiki and it is there as part of cleanup process.

 

[davidand]

in the newer version of i always have to execute rm -rf /etc/pve/nodes/<NODE Name> to remove it in the GUI. Also done this many times. Adding, removing and reusing nodes and readding them. Also did that when i renamed a node in my cluster because it created a new entry

It used to work for me in the 'older' versions, as well, I have had removed a couple of nodes in the past (usually HW upgrades / failures), with no issues. So it's new to me, too. Maybe the node that was being removed was still online in the past? I don't remember, but I'd say I did have HW failure when the machine was just plain dead.