Forgot to shutdown node before deleting it from cluster

[newbieProx]

Hi everyone,

I accidentally forgot to shut down the node before running the `pvecm delnode` command. Now I am facing issues with the cluster configuration. Could anyone guide me on how to resolve this?

Error: Killing node 2
closing file 'etc/pve/corosync/.conf.new.tmp.1272494' failed - Operation not permitted

the node still appears in the web GUI with a red icon, and some cluster configurations still reference this node.

Could anyone provide guidance on how to ensure that H2 is completely and cleanly removed from the cluster without breaking it or is there a way to restore and fix it.

Thanks in advance for your help!

 

[sw-omit]

First of all we'll probably need a bit more info:
How many nodes were and are there in your cluster, and if it was an even-numbered cluster, did you have a vote-deamon installed?
What is your plan with the deleted node, Do you want to keep using it with it's current setup, or re-install / scrap?
What is the output of pvecm status

 

[newbieProx]

Sure, there were 2 nodes in the cluster. I want to temporarily remove H2 from the cluster and swap drives, reinstall proxmox OS and then reconfigure to ZFS storage and rejoin the cluster.

Following are the current configurations in H1, which is the main node in cluster.


Cluster information
-------------------
Name: MC
Config Version: 2
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Mon Jul 15 11:27:33 2024
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 1.40
Quorate: No

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.10.0.1 (local)



Corosync.conf:
at /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: H1
nodeid: 1
quorum_votes: 1
ring0_addr: 10.10.0.1
}
node {
name: H2
nodeid: 2
quorum_votes: 1
ring0_addr: 10.10.0.2
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: MC
config_version: 2
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}


pveversion:
proxmox-ve: 8.1.0 (running kernel: 6.5.11-4-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.0.9
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
proxmox-kernel-6.5: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.4
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.9
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-1
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.2
pve-qemu-kvm: 8.1.2-4
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3

 

[sw-omit]

Since there are only 2 nodes in a cluster, you were "lucky" that you kept Quorum before, as it requires 3 or 2+a vote-deamon to properly function (is it requires "more then half good" to function, and currently you are at 50% working, so not more then 50%).

As it stands (and it says in the pvecm status, Quorate: No) you are currently not in Quorum, so can't do any changes.
To temporarily override this issue so you can continue working, run the command pvecm expected 1
Note that this is a temporary fix, and once you are back with 2 nods you should really consider either adding a third, or adding an (off-proxmox) device somewhere as a vote-deamon, for more info on that look here:
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[newbieProx]

Thank you so much for explaining it to me. I'm relatively new and learning everyday about the proxmox environment.

I would also request for further advise, is it safe to shutdown H2 and proceed with a fresh installation, without affecting anything in H1 node or entire cluster. Since H1 hosts critical VM's which I cannot afford to shutdown.

If the cluster isn't broken then is there a need to recreate a cluster from scratch? And yes i'm planning to join additional nodes to it as well and have a cluster of 5 proxmox host.

 

[sw-omit]

I would suggest to first shut down the H2 node, then remove it from the config [1] and then try to do any (safe) changes to your cluster, like for example trying to make a new (test) VM. As long as you can create the VM and it starts (doesn't need to install anything, starting to an iso is good enough) you should be good to go.
If you want to be on the extra-safe side, maybe rename / re-IP this node during the install to something else so any left-overs you might find later are clearly from that install, although if you follow the instructions to the letter, there shouldn't be anything left over.

Also, if you want to add that external vote-deamon for the times that you have an even number of votes, remember that you need to remove the vote-deamon BEFORE you add a new node to the cluster, then if you add a 4th one and are even once more, you can re-use the vote-deamon, just be sure to run the commands from node 3 or 4 (and install the apt-packages on 3 and 4).

That's how I did it myself during a migration, node 1 up, move VM's, node 2 up and added + vote-deamon added, move VM's, vote-deamon removed, node 3 up and added, move VM's, node 4 up and added + vote-deamon added, move VM's, vote-deamon removed, node 5 up and added.

[1] https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node