3.4 -> 4.0 Planning Input

[lweidig]

We just currently expanded a cluster from 2 nodes to 3 nodes on our 3.4 system. We have a mixture of KVM / VZ images running and were looking into the upgrade process. Since we need to have very little to no downtime based on the servers functions I do not see the standard upgrade procedure as a viable option for us. Some machines use local storage but most or on a SAN using iscsi. Therefore we are thinking of trying the following:

1. Migrate all of the servers on the new 3rd node to the other two.
2. Remove this node from the cluster.
3. Do a clean installation of this using the 4.0 install and wiping all of the disks.
4. EARLY AM backup a image, move to new system bring up the image. Do this for all nodes left on one of the two devices in the 3.4 cluster.
5. Once a second box has no running images, bring it down and perform a clean installation to 4.0 wiping all of the disks.
6. Bring it up and create a cluster with the two systems now running 4.0. Migrate some of the machines to this to ease load.
7. Repeat the last 3 steps for the final node that is no stand alone in 3.4.

We should now have the 3 systems running on 4.0 with a new cluster created and everything "fresh". Just looking to see if something is missing or if anybody has any additional "gotchas" that I might want to be aware of before starting down this path.

Thanks in advance!

 

[Richard]

Possible and should work.

But for the KVMs I see a similar way with minimum downtime (just shutdown and start, no special backup/restore necessary, even it is always recommended to have a backup for case of failures, faulty handling etc.)

1. Migrate virtual disks which are currently local to network storage

2. Save all configuration data /etc/pve/nodes/*/qemu-server/*.conf somewhere

3. Migrate VMs from the first node to be upgraded to one of the two others (step 1 in your list)

4. Remove this node from the cluster (step 2 in your list)

5. Do a clean installation of this using the 4.0 install on that node (step 3 in your list)

6. Restore the previously saved conf data in the newly installed node (and define networks, storages as well as they were before

7. Shutdown the VMs dedicated for the newly installed node (but currently running at one of the others) and start them immediately afterwards on the new node

8. Create then a new cluster

9. Perform steps 3-5 for the second node accordingly

10. Join the second node to the new cluster

11. Perform step 7 for the second node accordingly

12. Move the conf files of the VMs which are still running on the last old node to one of the two other node directories in the new cluster

13. Shutdown the VMs on that last old node and start them immediately afterwards on one of the two new nodes

14. 5. Do a clean installation of this using the 4.0 install on that last node

15. Join the last node to the new cluster

16. Migrate the VMs dedicated for the last node back to it

For containers you have to backup/restore/convert and adapt configuration since lxc is not fully compatible to OpenVZ.

 

[lweidig]

Ok, so hoping this helps others as they make the journey from 3.4 to 4.0. For us it has went pretty well, but there are a number of gotcha's that we will point out. First, the "plan" as originally stated and then expanded upon worked well. We were able to move both OpenVZ images and KVM guests over fairly cleanly. Here is what was not clean:

1. CentOS 5.11 OpenVZ containeers needed this fix:

http://forum.proxmox.com/threads/23921-Patch-Proxmox-4-0-CentOS-5-and-6-Containers-not-starting​

Also needed to have .conf file manually configured as the restore left a lot out​

2 .Ubuntu 10.04 OpenVZ containers needed this fix:

Edit /usr/share/perl5/PVE/LXC/Setup/Ubuntu.pm​

Add '10.04' => 1, # lucid LTS to $known_versions hash​

Update if:​

if ($version eq '10.04' || $version eq '12.04' || $version eq '14.04') {​

- Manually add ostype: ubuntu to .conf file and other changes​

3. LXC containers load averages are that of the host so if monitoring be prepared, this basically just ended up giving us a lot of alerts. OpenVZ seemed to have its own load averages.
4. Capabilities in LXC

This is still an unresolved issue for us. Please see the thread here and post any solutions:​

https://forum.proxmox.com/threads/23964-LXC-Capabilities​

5. FreeNAS 9.3.1 uses the new iSCSI implementation slated for FreeNAS 10 which is NOT compatible with ZFS over iSCSI and will NOT work!

Wasted a lot of time on this one.​

Does appear that somebody is working to get this added, but no timeframe I could find​

6. NO live migration for LXC containers and NO ability to use ZFS over iSCSI storage.

If you make the fist two changes BEFORE restoring the container the import process will be much more smooth and require less/no editing of the .conf files directly

The last one for us is the most critical. We run a cluster and would like to have the ability to live migrate the LXC containers as opposed to shutdown, move and restart. I see this on the roadmap and hope that it is SOON. Other than that we have our 3 node cluster completely upgraded and all containers / guests up and running.