Hi guys,
Just a reminder to future myself and to help others, how I migrated from 6.0-9 without restarting a single VM (well almost, for testing purposes I had to restart some).
As subject says I decided to move a production Proxmox 6.0-9 to 8.0.3.
System:
- Lenovo server 1 (4x 10Gbps net)
- Lenovo server 2 (4x 10Gbps net)
- Lenovo storage dual double controller (4x 10Gbps fiber)
- Synology NAS
Configuration is a 2 node cluster, multipath LVM over iSCSI to Lenovo storage and NFS to Synology NAS for backups. Servers are not over provisioned, a single one can handle a full load. So I used quite a lot of live migration to move VM's from one to another as needed when doing cloning and backuping.
Steps:
0. Read all the manuals and how to's. Once you are done, read again and again and again and check what went wrong when others did it
1. Use Clonezilla to clone all nodes to Synology and test if you can restore. I've live migrated VM's from on node to the other, clone the empty one and repeat on the other node.
2. Backup all VMs and test if backup works & make snapshots within the storage
3. Live migrate all VMs from node1 to node2
4. Reboot empty node & bump from 6.0-9 to 6.4-15 with apt upgrade
5. Manually upgrade all that is needed to be ready for Proxmox 7 by following official wiki at this link.
6. Reboot and test if everything is working
7. Migrate first VM back to node 1 and test: networking/vlans, disks and general performance
8. if OK, migrate the rest of VMs to node1
9. Repeat the proces from 4. on node 2.
10. Once both at 6.4-15, Clonezilla all nodes to Synology
11. Backup again all VMs
12. Download Proxmox 8 ISO and install to node 1 (I manually configured networking & iscsi, I presume copying the right cfg would work?)
13. Join node 1 with new IP to cluster and remove the old node 1.
14. Use a VM backup preferably Windows OS (tested Windows 7 & Windows Server 2019) as with Linux we do not expect problems and test on node 1 (now @ Proxmox 8) if there are changes of network or any other detected hardware)
15. If everything went right, at this point the cluster cosists of Proxmox 8 and 6.4-15
16. As there were no HW changes that would disrupt my VMs I decided to live migrate everything from Proxmox 6 to 8 all VMs
17. Live migration worked OK (transferred all Windows instances and Linux VMs), for testing purposes I live migrated back from Proxmox 8 to 6.4-15 a Windows 7 install and it worked, I can move it back and forth without interruptions and any detectable problems.
18. Repeat step 12 for node 2
19. Remove old nodes from cluster:
- pvecm expected 1 (override expected number of quorum votes - if needed)
- pvecm delnode <node name>
The following step is not necessary if you deleted the node from cluster, if not then this is your option, how to change the IP. I tested both options.
20. Return the original IP's to nodes by the following formula: https://forum.proxmox.com/threads/change-cluster-nodes-ip-addresses.33406/#post-321731
- Change IP assigned to vmbr0, add network settings for the new interface so cluster can communicate.
- Edit /etc/pve/corosync.conf with the new ip, and increment the config_version setting
- edit the /etc/hosts with the new ip value
- Reboot the node
- wait for the node to reboot and perform next node, until end of nodes.
Configurations that are not handled by the cluster like iscsid.conf and multipath.conf you can just copy from your backup, I tested it works, the same goes for /etc/network/interfaces. The longest thing that took me 2 days to figure out is why multipath works on one node and not on the other, well I forgot to repopulate /etc/multipath/wwids with multipath -a xxxxxxxxx.
I hope it helps to the future lonely rider. These are just general steps, I didn't want to make the post longer as needed. If there will be questions I do have detailed comments in my notes.
Kind regards,
Mitja
Just a reminder to future myself and to help others, how I migrated from 6.0-9 without restarting a single VM (well almost, for testing purposes I had to restart some).
As subject says I decided to move a production Proxmox 6.0-9 to 8.0.3.
System:
- Lenovo server 1 (4x 10Gbps net)
- Lenovo server 2 (4x 10Gbps net)
- Lenovo storage dual double controller (4x 10Gbps fiber)
- Synology NAS
Configuration is a 2 node cluster, multipath LVM over iSCSI to Lenovo storage and NFS to Synology NAS for backups. Servers are not over provisioned, a single one can handle a full load. So I used quite a lot of live migration to move VM's from one to another as needed when doing cloning and backuping.
Steps:
0. Read all the manuals and how to's. Once you are done, read again and again and again and check what went wrong when others did it
1. Use Clonezilla to clone all nodes to Synology and test if you can restore. I've live migrated VM's from on node to the other, clone the empty one and repeat on the other node.
2. Backup all VMs and test if backup works & make snapshots within the storage
3. Live migrate all VMs from node1 to node2
4. Reboot empty node & bump from 6.0-9 to 6.4-15 with apt upgrade
5. Manually upgrade all that is needed to be ready for Proxmox 7 by following official wiki at this link.
6. Reboot and test if everything is working
7. Migrate first VM back to node 1 and test: networking/vlans, disks and general performance
8. if OK, migrate the rest of VMs to node1
9. Repeat the proces from 4. on node 2.
10. Once both at 6.4-15, Clonezilla all nodes to Synology
11. Backup again all VMs
12. Download Proxmox 8 ISO and install to node 1 (I manually configured networking & iscsi, I presume copying the right cfg would work?)
13. Join node 1 with new IP to cluster and remove the old node 1.
14. Use a VM backup preferably Windows OS (tested Windows 7 & Windows Server 2019) as with Linux we do not expect problems and test on node 1 (now @ Proxmox 8) if there are changes of network or any other detected hardware)
15. If everything went right, at this point the cluster cosists of Proxmox 8 and 6.4-15
16. As there were no HW changes that would disrupt my VMs I decided to live migrate everything from Proxmox 6 to 8 all VMs
17. Live migration worked OK (transferred all Windows instances and Linux VMs), for testing purposes I live migrated back from Proxmox 8 to 6.4-15 a Windows 7 install and it worked, I can move it back and forth without interruptions and any detectable problems.
18. Repeat step 12 for node 2
19. Remove old nodes from cluster:
- pvecm expected 1 (override expected number of quorum votes - if needed)
- pvecm delnode <node name>
The following step is not necessary if you deleted the node from cluster, if not then this is your option, how to change the IP. I tested both options.
20. Return the original IP's to nodes by the following formula: https://forum.proxmox.com/threads/change-cluster-nodes-ip-addresses.33406/#post-321731
- Change IP assigned to vmbr0, add network settings for the new interface so cluster can communicate.
- Edit /etc/pve/corosync.conf with the new ip, and increment the config_version setting
- edit the /etc/hosts with the new ip value
- Reboot the node
- wait for the node to reboot and perform next node, until end of nodes.
Configurations that are not handled by the cluster like iscsid.conf and multipath.conf you can just copy from your backup, I tested it works, the same goes for /etc/network/interfaces. The longest thing that took me 2 days to figure out is why multipath works on one node and not on the other, well I forgot to repopulate /etc/multipath/wwids with multipath -a xxxxxxxxx.
I hope it helps to the future lonely rider. These are just general steps, I didn't want to make the post longer as needed. If there will be questions I do have detailed comments in my notes.
Kind regards,
Mitja