Correct/Official procedure to update a PVE7 Cluster with Ceph 16.2

zeuxprox

Renowned Member
Dec 10, 2014
89
5
73
Hi,

is there an official procedure to update a PVE7 Cluster with Ceph 16.2?

I have a Cluster with 5 nodes PVE 7.0.10 with Ceph 16.2.5. Up to now this is the procedure I have used (for example to update node 1):
Code:
1. Migrate all VMs on node 1 to others nodes
2. apt update
3. apt dist-upgrade
4. reboot

When node 1 is ready I wait until Ceph is OK and then I update node 2.
The question is: is this procedure right? Do you suggest to set, before starting the update, some flag on ceph like:
  • ceph osd set noscrub
  • ceph osd set nodeep-scrub
  • ceph osd set noout
?

Any advise is welcome...

thank you
 
Overall the procedure sounds good. Setting the OSD flags can be a good idea if you expect a node to not be back in time and prevent Ceph to rebalance/recover the data on the remaining nodes.

OSDs will be set to out automatically if they are not back within 10 minutes.

Also, when you live migrate VMs in the cluster, be aware that we do make sure that live migrations will work from older to newer PVE will work. Newer to older PVE can work, but don't rely on it.
 
Overall the procedure sounds good. Setting the OSD flags can be a good idea if you expect a node to not be back in time and prevent Ceph to rebalance/recover the data on the remaining nodes.

OSDs will be set to out automatically if they are not back within 10 minutes.

Also, when you live migrate VMs in the cluster, be aware that we do make sure that live migrations will work from older to newer PVE will work. Newer to older PVE can work, but don't rely on it.
Now the question is: before update node 2 do you advice to unset the OSD flags with
  • ceph osd unset noscrub
  • ceph osd unset nodeep-scrub
  • ceph osd unset noout
wait until Ceph is OK and then repeat the procedure as done for node 1 (set the OSD flags again, upgrade node 2 and then unset the OSD flags) ?

Thank you
 
Well, if the first node is up as expected, the Ceph health should also be okay and therefore you should be fine, leaving the flags on until you are done. If you do have some issues where a node might not come up again or if you have a faulty OSD that needs to be replaced, then you can consider unsetting these flags until the cluster is healthy again. Then set them again and proceed with the upgrade.
 
Well, if the first node is up as expected, the Ceph health should also be okay and therefore you should be fine, leaving the flags on until you are done. If you do have some issues where a node might not come up again or if you have a faulty OSD that needs to be replaced, then you can consider unsetting these flags until the cluster is healthy again. Then set them again and proceed with the upgrade.
Hi,
so if I understand correctly, you suggest to set the following Ceph flags:
  • ceph osd set noscrub
  • ceph osd set nodeep-scrub
  • ceph osd set noout
before starting the update of node 1 and to remove them with:
  • ceph osd unset noscrub
  • ceph osd unset nodeep-scrub
  • ceph osd unset noout
only when all nodes of PVE7 cluster are updated, right?

Thank you
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!