Update PVE, Ceph and PBS

[licenciamento]

Hi everyone, how are you?

Today we have 4 Proxmox clusters, 1 pure Ceph cluster, and 3 PBS clusters, where we are studying and assembling the upgrade plan: PVE from 8.x to 9.x, Ceph from 18.x to 19.x and subsequently to 20.x, and PBS from 3.x to 4.x.

In dc1-cluster01 we have 8 nodes, all with PVE version 8.3.0, Ceph has 6 nodes on version 18.2.4 and 2 nodes on version 18.2.7.

In dc1-cluster02 we have 5 nodes, all with PVE version 8.3.0, Ceph is all on version 18.2.7.

In dc1-cluster03 we have 8 nodes, all with PVE version 8.3.0, Ceph is all on version 18.2.7.

In dc2-cluster01 we have 5 We have 4 nodes with PVE version 8.4.16 and one node with version 8.4.14; all ceph nodes are at version 19.2.3.

In the pure ceph dc1-ceph(HDD) cluster, we have 5 nodes; all ceph nodes are at version 18.2.4.

The PBS nodes are: pbs001 at version 3.2.2, pbs002 at version 3.3.0, and pbs003 at version 3.4.8.

We checked the documentation for updating PVE from 8.x to 9.x, and it states that we must have ceph version 19.x. That is, we need to update ceph to 19.x first. The documentation for updating ceph from 18.x to 19.x states that before updating, it is recommended that PVE be at the latest available version of 8.x before updating to 19.x. In this case, we will have... First, update all nodes in all clusters to the latest 8.x release, and only after that update the ceph to 19.x, and subsequently update the PVE to 9.x. Would this be the correct approach?

Could you please advise on the best way to perform the update with minimal downtime, keeping in mind that all clusters are in production?

 

[SteveITS]

That is my understanding, though we haven't upgraded to 9 yet, and already have Ceph 19. Presumably your clusters with 8 nodes have a Qdevice? IIRC that doesn't have a separate upgrade necessary...? As I recall it's recommended to just upgrade nodes one at a time and try not to have a mixed version cluster for very long. I would move VMs off a node before upgrading it.

You can just upgrade PBS, PVE 8 and 9 will back up to PBS 3 or 4.

Having PVE on a current version before upgrade incorporates any features that the PVE9 upgrade expects to be there, and of course any bug and security fixes along the way.

 

[alexskysilk]

This can be done without any downtime. and yes, you want to make sure that the running ceph version is available on the next distro- so make sure before you start you upgrade your ceph to squid. This process is non disruptive and will not introduce downtime. see https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid for a step by step.

the procedure to upgrade the cluster is pretty straightforward. follow the below steps ONE NODE AT A TIME:
1.migrate all workload to other nodes. setting maintenance mode is recommended- see https://pve.proxmox.com/wiki/High_Availability#ha_manager_node_maintenance
2. follow the upgrade procedure (update repos, dist upgrade)
3. reboot. wait until node is back up. unset maintenance mode if set.
4. go to next node. repeat until entire cluster has been updated.
5. I wouldnt upgrade to tentacle just yet, but thats an available next step.