[SOLVED] Unable to rectify previous cluster

[cshill]

Hi Proxmox community,
I have done this before correctly but for some reason now I'm having a problem breaking up a cluster and rejoining afterwards.

I have:
proxmoxtest
proxmoxtest2
proxmoxtest3

I had them all joined correctly and it was working fine. However, I am doing different tests and ran out of static IPs and didn't need proxmoxtest. I shut it down and used it's static IP on another server with the same name. It looked like it failed to join then refreshed the page and I saw it had configurations but showing some servers going online and offline.
Eventually the main cluster pool it was joined to said, nope this isn't the server we had, this is a bad name resolution. I went ahead and rebooted all of the servers as I was expecting some services or config files needed to be updated. That didn't fix it unfortunately and now it is saying this is only a temporary failure in name resolution 500 on the main cluster group.
I thought maybe this was a failure in the process of joining so I tried to remove the node again and find the config files and delete them. I am told I cannot delete the directories in /etc/pve/nodes as I don't have the permissions, but I'm the root user.

Even if I was to redo the OS on proxmoxtest the other two servers in the cluster recognize that proxmoxtest has been removed when you run pvecm nodes but the node is clearly visible but offline in the cluster group.


Anyone have an idea why I can't delete the configuration directories as I have followed this doc before and no problem. https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[Kingneutron]

...How do you run out of static IPs?

Srsly, I have a pretty extensive homelab with VMs, laptops, Macs and I'm not even up to 250 ip's in the house, and I have 3 different network speeds. Although they are on different subnets. 1Gbit is standard 192.168.1/24, 2.5Gbit is on 172.16.25/24, and 10Gbit is on 172.16.10/24

AFAIK, you can't play musical chairs with changing things around and expect a refurb node to rejoin a cluster, it should be a fresh install

 

[cshill]

I am doing this at a large datacenter and was provided a certain amount of IPs.

As for the node I removed it then did a fresh install on a different server and joined it with that hostname and IP. Seems to not work correctly.

 

[Kingneutron]

IDK, try a different hostname? Feels like the cluster still knows about the previous config and it's confusing it

 

[cshill]

I figured it out. I tried to delete the directory in /etc/pve/nodes/*directory_name* and it wasn't deleting despite being root. I was able to install sudo which for some reason allowed it despite me being root. Once that was gone I was able to add the node correctly.

I did however learn that when you decide to strip a node from a cluster you should stop replication and HA as now I'm working through that.