[SOLVED] Node not connecting to cluster

[losarbitan]

Hello all,

Recently I downsized my 4 node cluster to just 2 nodes. I had some free IP's now and wanted to switch my secondary node (SystemB) from 10.11.12.108 to 10.11.12.104. Whilst my main node (SystemA) lives on 10.11.12.105. I did not read up on correct procedure for changing the IP of nodes and messed up a bit. I edited the /etc/hosts and /etc/network/interfaces file on SystemB and ran an ifreload -a to update the changes. I also changed the hostname from SystemB to System2. I then noticed the cluster issues. Subsequently I decided to revert these changes so that I could use proper procedure to edit the IP of a node. However, after reverting the changes back to the original state the nodes no longer seem to be able to connect to each other. Actually I tried this twice and the cluster issues only started on the second time of trying this, I do not remember exactly what changed between the first and second time I tried it. I think I only changed the /etc/hosts file the first time.

I have checked and the /etc/corosync/corosync.conf and /etc/pve/corosync.conf files are all identical to each other on both nodes. I have tried many of the proposed solutions that I could find on the forums here. I think I did not do anything lasting to my configs anyway. All permanent changes that I did are (I think):

1. Increased the quorum vote of SystemA from 1 to 2, I think I did not increment the config version, but this was before I had cluster issues (I think).
2. Upped the corosync conf files to a version number higher than previously tested (version: 7)
3. started up local mode and this created 2 pmxcfs processes (I think) that I now can not seem to get rid off when I run a killall $(pidof pmxcfs), this just returns the pid's followed by no process found.

Attached to the post are the config files I was talking about. Since I can only add 10 files the output of status commands are in the post.

Thanks in advance for any help!

 

[losarbitan]

I solved the issue. Guess I did not read up on Proxmox Cluster Filesystem properly.
https://pve.proxmox.com/mediawiki/index.php?title=Proxmox_Cluster_file_system_(pmxcfs)&redirect=no

This page gave me the idea to check for pmxcfs lockfile, after deleting this on SystemA and rebooting pve-cluster.service everything went back to normal. Now I am going to change the IP after doing some research on how to do it proper.