Q: Upgrade 3-node Cluster (PVE 2.3->3.2) Sanity check

[fortechitsolutions]

Hi,

I'm trying to plan for a possible Proxmox upgrade,
- cluster of 3 proxmox hosts (Intel modular server, proxmox cluster, have shared LVM storage from modular server chassis for VMs - kind of a "SAS attached SAN")
- works fine but version is starting to become concern, ie, upgrade to PVE 3.X will be nice.
- I have reviewed process docs as per PVE Wiki, https://pve.proxmox.com/wiki/Upgrade_from_2.3_to_3.0

It sounds quite clear cut, even for Cluster, ie,

• first PVE host: evacuate all VMs so nothing is running here (live/migrate to other cluster nodes)
• do the upgrade, reboot, migrate VMs back into the new 3.X PVE node
• rinse and repeat on other nodes

I don't have access to a dev test environment that mimics this. So that kind of stinks. I prefer to avoid 'horrible disruption' of the upgrade going poorly and having to do clean install, restore everything from cold backups, etc. as the amount of VM disk image space is not trivial (ie, ~10Tb LVM LUN which is mostly provisioned full of VM images running on this cluster).

I wanted to kind of put out a sanity test query,

• has anyone else running a Proxmox 2.X cluster -- done an upgrade using script as per the PVE Wiki link, https://pve.proxmox.com/wiki/Upgrade_from_2.3_to_3.0 - and was the experience a success ?
• wanted to confirm, inherent in this process - appears to be that we have temporarily in place a "Mixed Node" cluster, ie,
• after we upgrade node1, but have not yet updated node2,3 - we have a cluster with one PVE3.X node, and two x PVE2.X nodes.
• then we upgrade node2, have 2 nodes on PVE3 and 1 node on PVE2
• finally upgrade node3, we have all 3 nodes on PVE3, fun work is finished.

I had tested PVE upgrade scripts earlier in the year in a test environment, as follows,

• upgrade standalone PVE1.X to PVE2.X via script, reboot
• try then to upgrade this host via script, from PVE2 to PVE3.
• at this point the upgrade process seems to have failed, at least for me, and I ended up with a non-bootable box. I didn't end up testing more, being slightly concerned with how the process went.

• Clearly doing a clean install / burn down and restore-vm-from-backup method is "Safe" but has the minor downside, of fairly large downtime for your VMs with this approach. (ie, 10Tb of data takes a while to move over gig-ether, especially when moving a couple of times - off the old box into NFS storage, then back into 'new box' after clean install).
• Clearly another way to do it would be "just buy new hardware", stand up a new PVE3.X cluster, and then one at a time, power off VMs, archive them to NFS storage, restore them on the new PVE cluster, and happy days ensue. But this still has a series of 'shorter' outages (ie, each individual VM has an outage window based on how big its disk is / how long it takes to backup-then-restore .. via gig ether in this case .. the VM).

So, anyhow, I have rattled on more than long enough.

If anyone is able to give any comments or feedback it would really be greatly appreciated.

Thanks,

Tim

 

[tom]

the upgrade script from 2 to 3 is working great (as long as do not have custom packages installed). as long as you are familiar with apt, I do not expect issues. but of course, such a migration always need planning, involved downtime and of course, you need a plan B. (backup).

 

[fortechitsolutions]

Hi Tom, thanks for the followup to my query; greatly appreciated. I am glad to hear this Proxmox 2->3 upgrade method has a good baseline for 'expectation for success'. I'll plan for this suitably / with a "plan B" (full backups) ready just in case.

Thanks,

Tim

 

[jleg]

Hi Tom, thanks for the followup to my query; greatly appreciated. I am glad to hear this Proxmox 2->3 upgrade method has a good baseline for 'expectation for success'. I'll plan for this suitably / with a "plan B" (full backups) ready just in case.

Thanks,

Tim

i used this script a few times (~ 6 nodes/ 2 clusters) - and i got it *always* working afterwards. on the other hand, a always had to do some mannual fine tuning/fixing afterwards, but this was probably because of having installed some other packages...
"fixing" afair consisted of some manual apt dependency resolving, and i think something of sort "grub-install" was needed. i am no Debian/apt expert - and it was resolvable though...