Accidentally removed node from cluster now stuck

[paulmorabi]

Hi,

I'm running Proxmox 5.3.12 on two nodes (A & B). I had them in a cluster. I was toying around with some cluster commands and I accidentally removed node B from the cluster. I now can't start any VM's on node B because it is complaining about the quoram. On node A, I've been able to restart VM's by reducing the quoram to 1.

I tried rejoining the cluster from B to A. However, it says "* this host already contains virtual guests". I reluctantly tried to remove the VM's but that also doesn't work because there isn't quoram. I searched online and found that stopping pve-cluster and running "/usr/bin/pmxcfs -l" works in that I can start the VM's again.

Is there any way I can re-join the cluster? If I have to wipe node B and restart again, and if it's the same hostname, will it have issues rejoining the cluster? (as the node is greyed out on A)

Any help would be very appreciated!

 

[mira]

Please follow the documentation for separating a cluster node without reinstalling (https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall). This resets the 2nd node to a state where you can rejoin it if you want. Any VMs or CTs on the node you join (in this case node B) will be deleted on join.

 

[paulmorabi]

Thanks for replying. This has worked for node B.

Now I am trying to rejoin the cluster in node A. In the web UI on node A, cluster information says it's a standalone node with no cluster defined. Yet, the two cluster nodes (including the one I have removed) is still there. "Join Information" is greyed out so I can't rejoin.

pvecm status says:

Spoiler: pvecm status

root@pve:/etc/pve/nodes# pvecm status
Quorum information
------------------
Date: Sat Apr 6 08:40:32 2019
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 1/1012
Quorate: No
Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 1
Quorum: 2 Activity blocked
Flags:
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.10 (local)

and pvecm nodes:

Spoiler: pvecm nodes

root@pve:/etc/pve/nodes# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 192.168.1.10 (local)

I also tried to remove the old nodes data from /etc/pvecm/nodes but I a getting a permission denied.

EDIT: So I reblooted, deleted node B using "pvecm delnode" from node A and then was able to remove it from /etc/pvecm/nodes/. It's no longer displaying in the GUI. The only remaining issue is the LVM volume from node B is still displaying in node A. Any ideas on this?

 

[mira]

If it is still visible you have it in your storage config (/etc/pve/storage.cfg). Remove it from there and you should be fine. (Can be done via GUI 'Datacenter -> Storage' as well)

 

[paulmorabi]

Thank you, that worked!