Migrate 7-node PVE 4.4 cluster to a new subnet

[gradinaruvasile]

Hello,

We have a 7-node PVE 4.4 cluster and we have to move it into another subnet.
Can this be done by just changing the IP addresses in the web interface, shut down the machines, change the cables for the management interface (the links to the data storage are not changed) and start them up in the new subnet?

Will the cluster data survive like this?

If we changed the IP addresseses (and DNS entries), will it be needed to manually change the /etc/hosts files or any other settings?
Thank you.

 

[fortechitsolutions]

Hi, for what it is worth. I believe the bobcares link given was only to change the IP on a standalone proxmox host (not a cluster situation). I believe the official stance on Proxmox cluster, is that - you deploy it; and then you never change the name or IP of the hosts involved. Period. (as per, https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster, it tells us, "First, install the Proxmox VE on all nodes, see Installation. Make sure that each Proxmox VE node is installed with the final hostname and IP configuration. Changing the hostname and IP is not possible after cluster creation."

However. I believe ~there are various 'workaround' methods documented, unofficially, which may allow you to clobber and re-christen the nodes into a cluster with new hostnames/IPs. I suspect the general caveat applies, "have backups if the data is important" (ie, VM dumps on other disks, ie, handy-dandy NFS target) - in case things go horribly wrong and you lose everything, backups are nice to have especially when doing unsupported non-documented processes.

Digging in my notes from some hackery last year, I think something like this:

  188  reboot
  189  service pve-cluster stop
  190  pmxcfs -l
  191  rm /etc/corosync.conf
  192  rm /var/lib/pve-cluster/*
  193  service pve-cluster stop
  194  rm /etc/corosync/corosync.conf
  195  pmxcfs -l
  196  reboot
  197  pvecm status
  198  cd /etc/pve/
  199  ls -la
  200  cd /var/lib/pve-cluster/
  201  ls -la
  202  date
  203  rm ./*
  204  rm -rf ./*
  205  reboot

• possibly you want to take a backup of the rm -rf directories first before doing this.
• Possibly you want to do a bit more google digging first before trying this.
• Possibly you want to test this on a 2-node test-cluster about which you care *zero* if it is horribly corrupted and borked with this non-official process.

Generally speaking I believe the 'fun' with this is that the pve-cluster service and the 'shared cluster config mini-filesystem' does not like when you try to make changes in ways that are not intended / while things are still active. Hence you have to partially-break it; scrub some config; reboot; then finish the process of scrubbing the undesired cluster config; reboot to finish the messy work; and then finally you have a 'clean' host again which you could change hostname/IP if desired. And then from there you can proceed with "create a cluster for the 'so-called-first-time' in the new subnet.

this is all of course 100% unsupported hackery and you really should not trust anything so blatantly bad idea

but it might help. But really do test first on a throw-away mini-test 2-node environment. Just to try to save sadness later in the day.


Tim

 

[gradinaruvasile]

I was aware if the fact that you cannot change one node's config in a cluster.
The question was if we change the configs (changes will be applied after reboot) and then take the WHOLE cluster offline at once then start it up in the new location what would happen.
I could mess with config files even offline (booted from a Linux live cd), i suppose when all cluster is offline the cluster config is saved on static conf files that can be changed?

 

[fortechitsolutions]

Hi, I think I misunderstood what you said in your original post. I took your words, "move cluster to another subnet" to mean, "we will be changing the IP address of the proxmox nodes".

Broadly speaking my observation with proxmox cluster,
-- it is fussy if you create a proxmox cluster, and then once it is created, you attempt to change either IP or hostname of any cluster member, period. (ie, hence the unsupported 'how to de-christen a cluster member' steps in my post above).

-- the proxmox cluster config is a thing unto itself; and if you 'make a mess of the cluster' - the VMs and their config files which reside on the proxmox nodes are not impacted (lost, damaged, etc) by any mess that might be made of the clusters' operational (or non-operational) status. (ie, ability to manage all cluster members from a single proxmox node WebUI; ability to live-migrate VMs between proxmox nodes, etc - is separate from 'ability to get onto proxmox WebAdmin for <given node> and start or stop a VM running on that node.

-- so as a worst case scenario, even if you render the cluster 'broken' from a 'cluster management perspective' you still have an escape hatch, backup / copy your VM disks and config files to some other safe storage place; blow away the cluster and clean install it with desired configuration in terms of proxmox node IP addresses and hostnames; and then finally restore your VMs back onto proxmox nodes via copying etc.

VMs running on proxmox don't care what the Proxmox management network is / what subnet / etc. So long as they are bridged to proper NICs / with proper connectivity to their required network segment(s). Likewise the proxmox management / cluster does not care about the VMs or their operational state. (other than if you have HA configured and it is 'watching' the VMs to see if they are alive or not / and if it needs to intervene and boot up VMs on another cluster node in the case of a HA-proxmox-node-fail).

if you want to pre-deploy a cluster at one physical location, then power it off, disassemble everything after documenting all the connectivity (ie, what physical port is what subnet / role etc). Then box it all up ship to new location, cable it into service, power it all up, and it works. Since in this scenario you pre-deploy it with the IP / network / hostname config which mimics its final resting place deployment precisely. Or in your case it sounds like you are leaving the gear physically in one location, and wish to cut over proxmox node IPs from one LAN segment to another (different physical or VLAN connectivity to your management interface on proxmox nodes?)

anyhow. If I am still misunderstanding what you are doing, and this is not relevant, sorry for my trouble following what you are trying to do.


Tim

 

[gradinaruvasile]

We need to change all IP addresses. I was thinking maybe we get away with it if we change them at once, but since i decided it is not worth the risk of getting a non functional cluster (we need HA, live migration, one-node management) and unscheduled vm downtime.
So we will take the safer route and do the migration one VM at a time (we have a new server that is not yet installed, that will be used as the first node in the new cluster).

 

[fortechitsolutions]

OK - all clear! For sure there is downtime involved with multiple reboots to de-christen the proxmox cluster and then change IPs and then re-create a 'new' cluster on the same nodes. Strictly speaking it can be done fairly quickly (slowest step is the speed of reboot on your server, which probably needs to happen twice).

Definitely doing a per-VM backup:restore from old to new is a clean way to cut things over, if the backup-move-restore cycle is sufficiently fast for your downtime needs. (ie, staggered outages for each Vm as it is migrated).

Definitely is a safe route to do this.

Tim

 

[gradinaruvasile]

The older "servers" are in fact Optiplex desktops with I7 CPUS so reboot speed is not an issue