Correct/Official procedure to update a PVE7 Cluster with Ceph 16.2

[zeuxprox]

Hi,

is there an official procedure to update a PVE7 Cluster with Ceph 16.2?

I have a Cluster with 5 nodes PVE 7.0.10 with Ceph 16.2.5. Up to now this is the procedure I have used (for example to update node 1):

1. Migrate all VMs on node 1 to others nodes
2. apt update
3. apt dist-upgrade
4. reboot

When node 1 is ready I wait until Ceph is OK and then I update node 2.
The question is: is this procedure right? Do you suggest to set, before starting the update, some flag on ceph like:

• ceph osd set noscrub
• ceph osd set nodeep-scrub
• ceph osd set noout

?

Any advise is welcome...

thank you

 

[aaron]

Overall the procedure sounds good. Setting the OSD flags can be a good idea if you expect a node to not be back in time and prevent Ceph to rebalance/recover the data on the remaining nodes.

OSDs will be set to out automatically if they are not back within 10 minutes.

Also, when you live migrate VMs in the cluster, be aware that we do make sure that live migrations will work from older to newer PVE will work. Newer to older PVE can work, but don't rely on it.

 

[zeuxprox]

Overall the procedure sounds good. Setting the OSD flags can be a good idea if you expect a node to not be back in time and prevent Ceph to rebalance/recover the data on the remaining nodes.

OSDs will be set to out automatically if they are not back within 10 minutes.

Also, when you live migrate VMs in the cluster, be aware that we do make sure that live migrations will work from older to newer PVE will work. Newer to older PVE can work, but don't rely on it.

Now the question is: before update node 2 do you advice to unset the OSD flags with

• ceph osd unset noscrub
• ceph osd unset nodeep-scrub
• ceph osd unset noout

wait until Ceph is OK and then repeat the procedure as done for node 1 (set the OSD flags again, upgrade node 2 and then unset the OSD flags) ?

Thank you

 

[aaron]

Well, if the first node is up as expected, the Ceph health should also be okay and therefore you should be fine, leaving the flags on until you are done. If you do have some issues where a node might not come up again or if you have a faulty OSD that needs to be replaced, then you can consider unsetting these flags until the cluster is healthy again. Then set them again and proceed with the upgrade.

 

[zeuxprox]

Well, if the first node is up as expected, the Ceph health should also be okay and therefore you should be fine, leaving the flags on until you are done. If you do have some issues where a node might not come up again or if you have a faulty OSD that needs to be replaced, then you can consider unsetting these flags until the cluster is healthy again. Then set them again and proceed with the upgrade.

Hi,
so if I understand correctly, you suggest to set the following Ceph flags:

• ceph osd set noscrub
• ceph osd set nodeep-scrub
• ceph osd set noout

before starting the update of node 1 and to remove them with:

• ceph osd unset noscrub
• ceph osd unset nodeep-scrub
• ceph osd unset noout

only when all nodes of PVE7 cluster are updated, right?

Thank you

 

[t.lamprecht]

is there an official procedure to update a PVE7 Cluster with Ceph 16.2?

Just for the record, those are the official how-tos to upgrade from PVE 6.4 to PVE 7.x and Ceph Nautilus to Octopus, respectively:
https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0
https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus