Shutting down the cluster with ceph

[brucexx]

Any advice on powering down two clusters ?

I have a 6 node closure on 8.3.1 (Enterprise repo) with 50-65 VMs pre node. Storage is a separate 5 node Proxmox cluster with Ceph same 8.3.1 version - this cluster is not running nay VMs.

We have to power them down as part of a maintenance. The plan is to:

1. Disable HA , watchdog.
2. Disable power on boot.
3. Shutdown all VMs
4. Update cluster to latest 8.x and shutdown.
5. Update and shutdown Ceph cluster to latest 8.x. with ceph osd set noout ceph osd set norecover ceph osd set norebalance ceph osd set nobackfill.

After the maintenance concludes:

1. Power up Ceph first with osd unset flags and check Ceph status.
2. Power up PVE clusters.
2. We plan to reconfigure corosync and add ring 2 (currently on bonded interfaces), which have been stable but apparently multiple rings are preferred, we are still debating it.
3. Enable HA and watchdog.
3. Once both clusters are running manually start VMs and re-enable start at boot.

Is there anything I am missing ?

Thank you

 

[SteveITS]

Double check https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_shutdown

 

[Francois Baillargeon]

It is preferable to have two rings as a misconfigured switching equipment on a bond might make the link down or flap for no reason.


Imagine a failing switch with one of two members of a bonded link dropping say, 50% of the traffic on a bonded leg, corosync has more chance of misbehaving.

Having two separate physical links avoid that.

Just make sure to get a good copy of the config file and increment it.

Good luch with the maintenance!

 

[brucexx]

one more thing, I need to remove an aging node and add a new node that is identical (hardware wise) as the 5 existing nodes. I would like to do that a week ahead of maintenance so it is one less thing to do. How much differences between packages is acceptable ? My current cluster is on:

proxmox-ve: 8.3.0 (running kernel: 6.8.12-4-pve)
pve-manager: 8.3.1 (running version: 8.3.1/fb48e850ef9dde27)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-4
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 18.2.4-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.0
libpve-storage-perl: 8.3.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.2
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-1
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.2
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

can I just go all the way and update it to the latest or install 8.3.1-ISO with packages very close to the cluster I am running now. I did some testing with node A being the current running cluster:

Package	This node (8.3.1 ISO)	Node A
pve-manager	8.3.0	8.3.1
ceph-fuse	18.2.8-pve1	18.2.4-pve3
libpve-storage-perl	8.2.9	8.3.1
proxmox-backup-client	3.2.9-1	3.3.2-1
proxmox-backup-file-restore	3.2.9-1	3.3.2-2
proxmox-widget-toolkit	4.3.1	4.3.3
pve-i18n	3.3.1	3.3.2
qemu-server	8.3.0	8.3.2

Everything else is identical, including the full cluster stack (corosync 3.1.7-pve3, libknet1 1.28-pve1, pve-cluster 8.0.10, pve-ha-manager 4.0.6) and the running kernel 6.8.12-4-pve.

Let me know if this seems feasible, thank you

 

[Francois Baillargeon]

Rule of thumb is to be as close as possible for all nodes in your clusters, your idea of upgrading it to be as near as possible is good.