[SOLVED] PVE/Ceph cluster: Recommendations for updating and restarting

[sherminator]

Hi there,

today we like to install updates on our three nodes of our PVE/Ceph cluster which need restarts.

# apt list --upgradable
Listing... Done
ceph-base/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-common/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-fuse/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-mds/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-mgr/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-mon/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph-osd/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
ceph/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
git-man/stable 1:2.20.1-2+deb10u1 all [upgradable from: 1:2.20.1-2]
git/stable 1:2.20.1-2+deb10u1 amd64 [upgradable from: 1:2.20.1-2]
ifupdown/stable 0.8.35+pve1 amd64 [upgradable from: 0.8.35]
libcephfs2/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
libidn2-0/stable 2.0.5-1+deb10u1 amd64 [upgradable from: 2.0.5-1]
libnss3/stable 2:3.42.1-1+deb10u2 amd64 [upgradable from: 2:3.42.1-1+deb10u1]
libpve-access-control/stable 6.0-5 amd64 [upgradable from: 6.0-3]
libpve-common-perl/stable 6.0-10 all [upgradable from: 6.0-7]
libpve-guest-common-perl/stable 3.0-3 all [upgradable from: 3.0-2]
libpve-storage-perl/stable 6.1-3 all [upgradable from: 6.0-9]
librados2/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
libradosstriper1/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
librbd1/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
librgw2/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
libsasl2-2/stable 2.1.27+dfsg-1+deb10u1 amd64 [upgradable from: 2.1.27+dfsg-1]
libsasl2-modules-db/stable 2.1.27+dfsg-1+deb10u1 amd64 [upgradable from: 2.1.27+dfsg-1]
proxmox-ve/stable 6.1-2 all [upgradable from: 6.0-2]
proxmox-widget-toolkit/stable 2.1-2 all [upgradable from: 2.0-8]
pve-cluster/stable 6.1-3 amd64 [upgradable from: 6.0-7]
pve-container/stable 3.0-16 all [upgradable from: 3.0-10]
pve-docs/stable 6.1-3 all [upgradable from: 6.0-8]
pve-edk2-firmware/stable 2.20191127-1 all [upgradable from: 2.20190614-1]
pve-firewall/stable 4.0-9 amd64 [upgradable from: 4.0-7]
pve-ha-manager/stable 3.0-8 amd64 [upgradable from: 3.0-2]
pve-kernel-helper/stable 6.1-1 all [upgradable from: 6.0-12]
pve-manager/stable 6.1-5 amd64 [upgradable from: 6.0-11]
pve-qemu-kvm/stable 4.1.1-2 amd64 [upgradable from: 4.0.1-5]
python-apt-common/stable 1.8.4.1 all [upgradable from: 1.8.4]
python-apt/stable 1.8.4.1 amd64 [upgradable from: 1.8.4]
python-ceph-argparse/stable 14.2.6-pve1 all [upgradable from: 14.2.4-pve1]
python-cephfs/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
python-rados/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
python-rbd/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
python-rgw/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]
python3-apt/stable 1.8.4.1 amd64 [upgradable from: 1.8.4]
qemu-server/stable 6.1-4 amd64 [upgradable from: 6.0-13]
smartmontools/stable 7.1-pve2 amd64 [upgradable from: 7.0-pve2]

What's the recommended order of the required steps?

Should we
1. install updates on the first node, restart this one and do so with node 2 and 3?
2. install updates on all three nodes and then restart one node by one?
3. do something completely different?

Thanks and greets
Stephan

 

[Alwin]

This is up to you. But I would go with one node at time. If something goes wrong with the update on that node, you still have the rest of the cluster working.

ceph/stable 14.2.6-pve1 amd64 [upgradable from: 14.2.4-pve1]

In any case, you should do the update.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/X6TNSDQK5DVKO6XFJW3DMJAJV63PLDYM/

 

[sherminator]

Hey Alwin,

thanks! Indeed we did it this way:

1. migrate all VMs away from node1 (I love "bulk migration" over a dedicated 10 GBit/s-NIC)
2. install updates
3. restart
4. wait until all PGs are "active+clean"
5. do the same with node2 and node3

Greets
Stephan