Proxmox Cluster Upgrade from 6.4 to 7.3 (using Remote Ceph Storage)

[dpl]

Hello all,

We are currently running a Proxmox cluster consisting of 12 nodes with PVE 6.4 (latest patches) and local ZFS storage.

In the future, the PVE cluster will be fully connected using Ceph storage.

To migrate the infrastructure to the new PVE version 7.3, we wanted to use our external Ceph storage to offload the VM data (separate PVE 7.3 cluster with Ceph Quincy).
Since the RBD integration of the Quincy Ceph does not work on the current, old Proxmox 6.4 nodes with version "luminous", we wanted to exchange the data via a 7.3 node as a migration path.

So we would have next to the 12x 6.4 Nodes:
1. Add a thirteenth Node to the cluster
2. Upgrade this Node from 6.4 to 7.3
3. Mount the external Ceph storage only for Node 13
4. Move all VMs one by one from the twelve cluster note (to the local storage) to node 13
5. Move the VM data from the local storage to the shared Ceph storage
6. Upgrade all old nodes from 6.4 to 7.3
7. Share the Ceph storage with all other nodes
8. Distribute VM profiles back to the respective original nodes
9. Start the VMs from the shared Ceph storage on the respective upgraded nodes

Is this a reasonable, feasible approach?

 

[Richard]

Hello all,

We are currently running a Proxmox cluster consisting of 12 nodes with PVE 6.4 (latest patches) and local ZFS storage.

In the future, the PVE cluster will be fully connected using Ceph storage.

To migrate the infrastructure to the new PVE version 7.3, we wanted to use our external Ceph storage to offload the VM data (separate PVE 7.3 cluster with Ceph Quincy).
Since the RBD integration of the Quincy Ceph does not work on the current, old Proxmox 6.4 nodes with version "luminous", we wanted to exchange the data via a 7.3 node as a migration path.

So we would have next to the 12x 6.4 Nodes:
1. Add a thirteenth Node to the cluster
2. Upgrade this Node from 6.4 to 7.3
3. Mount the external Ceph storage only for Node 13
4. Move all VMs one by one from the twelve cluster note (to the local storage) to node 13
5. Move the VM data from the local storage to the shared Ceph storage
6. Upgrade all old nodes from 6.4 to 7.3
7. Share the Ceph storage with all other nodes
8. Distribute VM profiles back to the respective original nodes
9. Start the VMs from the shared Ceph storage on the respective upgraded nodes

Is this a reasonable, feasible approach?

In principle yes, but step 8+9 can be done by a simple "migrate VM".
Also be careful to not overlod node13 when it hosts VMs from all 12 other nodes. Proceeding more gently would be to start with node1 for this process, when it's completely finished node2 etc.

 

[dpl]

Running the migrated VMs from the 12 nodes all on the thirteenth node is not intended.

The thirteenth node is only used to take over the "VM (KVM) profile settings" and "VM (block) data" in the meantime, so that a data transfer (to CEPH) from the local storage (ZFS) of the nodes takes place.

Background to the thread: https://forum.proxmox.com/threads/can-i-mix-promxox-6-and-proxmox-7-in-the-same-cluster.92214/

To run a "mixed" Proxmox Cluster temporarily to implement a PVE infrastructure upgrade from 6.4 to 7.3.

 

[dpl]

Hello Richard,

We have successfully completed the infrastructure upgrade.
GPU passthrough also works so far on the 10 machines under kernel 5.15

We only had problems starting UEFI VMs with Ceph storage for a short time.
The error "rbd_cache_policy=writeback: invalid conf option rbd_cache_policy: No such file or directory" was fixed
by installing Ceph Quincy on all nodes (without setup).
By this upgrade of librbd the error disappeared.

see also: https://forum.proxmox.com/threads/rbd_cache_policy-error-no-such-file-or-directory.99438/