Changing the IP address and subnet in all cluster nodes

[jsabater]

Good evening everyone!

I have a production cluster with 5 nodes. All of the nodes have one network card with:

• A dedicated public IP address on eno1.
• A vmbr4001 bridge for the public subnet of the guests (LXC).
• A vmbr4002 bridge for the private subnet of the guests (LXC).
• A vmbr4003 bridge for the private subnet of the hosts (nodes) with an assigned IP address (e.g. 192.168.1.11)

I need to change the private IP addresses of all the nodes from the 192.168.1.0/24 to the 10.0.0.0/8 range. I have found this very nice guide. Basically what I would be doing is:

1. Stop the cluster: systemctl stop pve-cluster corosync on each node.
2. Locally mount the filesystem on each node.
3. Edit the files /etc/hosts and /etc/pve/priv/known_hosts to match the new subnet.
4. Edit /etc/pve/corosync.conf on each node and modify the IP addresses of all the nodes present there (from 192.168.1.0/24 to 10.0.0.0/8), incrementing the config_version to the same value on all nodes (e.g. from 8 to 9).
5. Reboot all nodes.

This is, as you all may imagine, a very sensitive procedure.

Questions:

1. Would it be better to reboot the nodes one by one, or reboot them all at once?
2. Is this procedure up to date? Anything missing?
3. Any hints, recommendations or musts beyond what is described above?

Thanks in advance.

 

[floh8]

A vmbr4003 bridge for the private subnet of the hosts (nodes) with an assigned IP address (e.g. 192.168.1.11)

This is your cluster network?

 

[jsabater]

This is your cluster network?

Yes, this is the private network for the hosts/nodes of the cluster to talk among themselves.

 

[floh8]

by changing the cluster config you have to be very careful. The howto you linked seems to be very handful. I would it a try. Best is to have Host backups for rollback. Also, I would not start immediately the VMs after change. Maybe the staff can give an extra hint.

 

[hvanmegen]

I'm also curious if this can do the same as I am in the same situation but with 6 nodes... I'm switching over from one smaller subnet to another larger subnet in an attempt to simplify our network structure

 

[ctrlbrk]

I'm also curious if this can do the same as I am in the same situation but with 6 nodes... I'm switching over from one smaller subnet to another larger subnet in an attempt to simplify our network structure

Did it work as laid out in the guide? I am in production, remote from datacenter, and need to modify our cluster home network to a new subnet.

 

[jsabater]

I ended up not attempting to perform the change. For now I am holding with what I have (although I don't have much room left) and I plan on creating a new cluster from scratch, as it will also serve to overcome the 32 MAC addresses limitation per NIC that Hetzner imposes on their vSwitches (I plan on connecting the new servers with hardware switches).

Please let me know how it goes, in case you decide to attempt it.

 

[DJFriar]

I did perform the changes as listed in the guide (the first section as I was changing subnets). While the IP change went through fine, in hindsight I should have done something to stop HA migrations as while I was changing the second node everything migrated to the first node and I am now struggling to recover from that.

In all fairness, my struggles are likely due to not quite getting the cluster storage configured right in the first place, and not related to the instructions in the guide linked above; but I would strongly advise pausing the HA service somehow before doing these steps (as a newb I assume that it is possible to do so, just not sure how).

 

[Rxunique]

I did perform the changes as listed in the guide (the first section as I was changing subnets). While the IP change went through fine, in hindsight I should have done something to stop HA migrations as while I was changing the second node everything migrated to the first node and I am now struggling to recover from that.

In all fairness, my struggles are likely due to not quite getting the cluster storage configured right in the first place, and not related to the instructions in the guide linked above; but I would strongly advise pausing the HA service somehow before doing these steps (as a newb I assume that it is possible to do so, just not sure how).

How long did you have to wait after rebooting the changed node for corosync to propagate through other nodes?

I followed the same guide but after rebooting node1 it separated from cluster and I'm stuck trying get the other nodes to accept new IP.

Or did you changed all files on all nodes in one hit? That sounds risky.

 

[jsabater]

Or did you changed all files on all nodes in one hit? That sounds risky.

Unless I have misunderstood you, yes, you would have to change all the IP addresses at once. Otherwise, they would not be able to talk to each other.