Changing the IP address and subnet in all cluster nodes

J

jsabater

Member
Oct 25, 2021
130
14
23
49
Palma, Mallorca, Spain
Good evening everyone!

I have a production cluster with 5 nodes. All of the nodes have one network card with:
  • A dedicated public IP address on eno1.
  • A vmbr4001 bridge for the public subnet of the guests (LXC).
  • A vmbr4002 bridge for the private subnet of the guests (LXC).
  • A vmbr4003 bridge for the private subnet of the hosts (nodes) with an assigned IP address (e.g. 192.168.1.11)
I need to change the private IP addresses of all the nodes from the 192.168.1.0/24 to the 10.0.0.0/8 range. I have found this very nice guide. Basically what I would be doing is:
  1. Stop the cluster: systemctl stop pve-cluster corosync on each node.
  2. Locally mount the filesystem on each node.
  3. Edit the files /etc/hosts and /etc/pve/priv/known_hosts to match the new subnet.
  4. Edit /etc/pve/corosync.conf on each node and modify the IP addresses of all the nodes present there (from 192.168.1.0/24 to 10.0.0.0/8), incrementing the config_version to the same value on all nodes (e.g. from 8 to 9).
  5. Reboot all nodes.
This is, as you all may imagine, a very sensitive procedure.

Questions:
  1. Would it be better to reboot the nodes one by one, or reboot them all at once?
  2. Is this procedure up to date? Anything missing?
  3. Any hints, recommendations or musts beyond what is described above?
Thanks in advance.
 
by changing the cluster config you have to be very careful. The howto you linked seems to be very handful. I would it a try. Best is to have Host backups for rollback. Also, I would not start immediately the VMs after change. Maybe the staff can give an extra hint.
 
hvanmegen said:
I'm also curious if this can do the same as I am in the same situation but with 6 nodes... I'm switching over from one smaller subnet to another larger subnet in an attempt to simplify our network structure
Click to expand...
Did it work as laid out in the guide? I am in production, remote from datacenter, and need to modify our cluster home network to a new subnet.
 
I ended up not attempting to perform the change. For now I am holding with what I have (although I don't have much room left) and I plan on creating a new cluster from scratch, as it will also serve to overcome the 32 MAC addresses limitation per NIC that Hetzner imposes on their vSwitches (I plan on connecting the new servers with hardware switches).

Please let me know how it goes, in case you decide to attempt it.
 
I did perform the changes as listed in the guide (the first section as I was changing subnets). While the IP change went through fine, in hindsight I should have done something to stop HA migrations as while I was changing the second node everything migrated to the first node and I am now struggling to recover from that.

In all fairness, my struggles are likely due to not quite getting the cluster storage configured right in the first place, and not related to the instructions in the guide linked above; but I would strongly advise pausing the HA service somehow before doing these steps (as a newb I assume that it is possible to do so, just not sure how).
 
DJFriar said:
I did perform the changes as listed in the guide (the first section as I was changing subnets). While the IP change went through fine, in hindsight I should have done something to stop HA migrations as while I was changing the second node everything migrated to the first node and I am now struggling to recover from that.

In all fairness, my struggles are likely due to not quite getting the cluster storage configured right in the first place, and not related to the instructions in the guide linked above; but I would strongly advise pausing the HA service somehow before doing these steps (as a newb I assume that it is possible to do so, just not sure how).
Click to expand...
How long did you have to wait after rebooting the changed node for corosync to propagate through other nodes?

I followed the same guide but after rebooting node1 it separated from cluster and I'm stuck trying get the other nodes to accept new IP.

Or did you changed all files on all nodes in one hit? That sounds risky.
 
You must log in or register to reply here.
Top Bottom