Changing the IP address of one node in a cluster

[bea]

Hello.

I have a cluster of nodes A, B and C. I wanted to change B's IP address.

On node B I did:

systemctl stop pve-cluster
systemctl stop corosync
pmxcfs -l

Then, on node B too, I changed its IP address on
/etc/network/interfaces
/etc/hosts
/etc/pve/corosync.conf
And incremented the config version on that last file

Then reboot

Then the PVE GUI showed a red sign on node B and question marks on its VMs. I tried unsuccessfully different things to solve it.

On A and C, pvecm status does not show the new config version number of B. They were not updated and still show the old number.
On A and C pvecm nodes only shows 2 nodes: A and C.

What did I do wrong?
How could I troubleshoot it?

Thank you.

 

[waltar]

Changing an ip of a pve node should never be done ! If you need to empty the node from vm/lxc, de-join the node, reinstall the node with new ip and re-join new node to pve cluster again.

 

[bea]

I did not know.
I will go for your suggestion, it's not quick, but it seems clean and easy (I hope so
Thank you!

 

[bea]

In order to do de-join and rejoin, I am trying to follow the official Proxmox documentation but that guide does not work in the situation I am, I guess it would work with a healthy cluster.

I undid the changes explained on my first post, then I reboot the node, but still the node does not get back into the cluster.

Despite having an unhealthy cluster, I tried to follow the documentation. I powered off node B. On node A I run pvecm delnode B but I get:
Node/IP: B is not a known host of the cluster

So, I'm stack, now. The cluster does not accept node B anymore and I cannot delete node B from the cluster.

What should I do?

Thank you.

 

[UdoB]

What should I do?

Slow down.

Your Nodes A + C are still okay, right?

1)
First make a current backup of /etc on both nodes A + C. If you have a systematic backup approach --> use it. Otherwise run something like cp -av /etc /var/backups/copy-of-etc-with-damaged-node-b. Sidenote: this includes the content of /etc/pve which is a fuse mount-point of the PVE configuration database as normal files.

2)
To confirm the status of these two nodes: post the output of pvecm status of one of them.

Depending on the output of you need to clean up the situation.

Anything which belonged to B inside of /etc/pve should get deleted. Additionally check /etc/hosts on A+C and remove everything B.

3)
On B you need to follow "Separate a Node Without Reinstalling" - https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

Everything belonging to A or C in /etc/* should get removed.

B does contain VMs, right? Freshly joining this node to the cluster A/C is a separate task. You should find some hints here in the forum on how to do that.


Disclaimer: EVERYTHING of the above is dangerous and not "normal" - you completely operate outside the usual parameters! Also: I have never done that, so take my statements with a grain of salt...

 

[bea]

Your Nodes A + C are still okay, right?

Yes.

post the output of pvecm status of one of them.

root@A:~# pvecm status 
Cluster information 
------------------- 
Name:             A 
Config Version:   6 
Transport:        knet 
Secure auth:      on 
 
Quorum information 
------------------ 
Date:             Thu Feb  6 18:16:04 2025 
Quorum provider:  corosync_votequorum 
Nodes:            2 
Node ID:          0x00000002 
Ring ID:          2.1a5 
Quorate:          Yes 
 
Votequorum information 
---------------------- 
Expected votes:   2 
Highest expected: 2 
Total votes:      2 
Quorum:           2   
Flags:            Quorate  
 
Membership information 
---------------------- 
    Nodeid      Votes Name 
0x00000002          1 192.168.1.101 (local) 
0x00000003          1 192.168.1.103 
root@A:~#

Being
A: 192.168.1.101
B: 192.168.1.102
C: 192.168.1.103

Depending on the output of you need to clean up the situation.

With the output above, is it ok to proceed with your suggestions?

 

[alexskysilk]

At this stage, you should just reinstall node b from scratch and add it to the cluster as a new node.

 

[bea]

But I still have not done any of the suggestions on post 5. I mean the cluster is still missing B, I can see it on the web-GUI. On A and C I have not removed anything.

I guess I should proceed with the steps suggested on post 5.

Thanks for following-up

 

[alexskysilk]

But I still have not done any of the suggestions on post 5

With all respect to @UdoB none of that is actually necessary. Once you've removed the node from the cluster, all you need to do is WIPE it and reinstall pve from scratch, and add it back to the cluster as if its a new node- because it is.

oh just one more thing- do NOT NAME IT THE SAME as the original name; it can cause signature issues.

 

[bea]

Ok, that sounds like a much safer thing. I don't feel like changing things on A and C unless it is a safe procedure, because A and B are working ok now .

So, with your suggestion I will have a 4-node cluster (A, B, C, D) with one node (B) missing.

Is there any safe way to remove that missing node (B)?

If there is no easy and safe way I guess for the time being I can live with that zombie node on the GUI with its red sign on it.

 

[alexskysilk]

Is there any safe way to remove that missing node (B)?

answer-

On B you need to follow "Separate a Node Without Reinstalling" - https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

tldr-
perform from any surviving node:
1. pvecm delnode "nodeb"
2. rm -rf /etc/pve/nodes/nodeb
3. sed -i '/nodeb/d' /etc/pve/.members

if you're not sure what your nodes are named, pvecm nodes will show you.

 

[UdoB]

With the output above, is it ok to proceed with your suggestions?

Yes. A+C did confirm that B is gone.

If it is okay for you to reinstall B then follow @alexskysilk advice. This is definitely the cleanest approach! Does B contain anything important? VMs without backups?

Btw:

With all respect to @UdoB none of that is actually necessary.

No problem; I just prefer smaller steps and to re-validate the situation.

So, @bea, how shall we procede?

 

[bea]

Yeees, thank you @alexskysilk !!

The webGUI looks clear now! Node B is gone!

all you need to do is WIPE it and reinstall pve from scratch

Now I am going to reinstall old node B and give it a new name.
Old node B had a ZFS mirror, and now, when installing from scratch, will choose again ZFS mirror.
Do I need to wipe it if I'm installing from scratch?
If so, why?

 

[UdoB]

Do I need to wipe it if I'm installing from scratch?

No, probably not.

The installation will take the two disks you give it inside of the setup process. It will create a fresh partition table and a clean ZFS pool.

 

[bea]

Thank you. I reinstalled from scratch (without previous wipe) and I joined the new node to the cluster.

The cluster looks ok but one thing: the new node shows a grey question mark on the PBS. This PBS was configured as storage on the datacenter level and it was shown by every node. It is still shown by every node, but the new node shows it with a grey question mark.

I rebooted the node. But nothing changed.

 

[bea]

I found why I got those grey question marks on the new node. The Boot Mode on the two healthy nodes on the cluster was EFI, while on the new node was Legacy BIOS. I did again a fresh install on that node to install with EFI and everything seems to be solved now : )

Thank you @waltar, @UdoB and @alexskysilk !