Change 3 Node Ceph Cluster to 1 Node

[mohnewald]

Hello
i have a old Proxmox 4.4 Cluster and would like to "reinstall" it.

My goal is to change that 3-Node Cluster with ceph storage to a Single-Node to get 2 free for a fresh install.

I guess i just change the quorum to 1 and then migrate all VMS to that one node. Then turn off the two remaining (emtpy) nodes and set ceph size/min_size to 1?

I that the way to do it?

Thanks,
Mario

 

[aaron]

You will have zero redundancy this way.

If your goal is to upgrade to the latest version you can upgrade in place to PVE 5.x and then to 6.x. Both steps are supported and documented:
https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0
https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0

If you really want to reinstall your nodes but keep the cluster, go the upgrade way and then reinstall them one by one afterward. This way you will keep redundancy because you will always have 2 nodes running.

 

[Alwin]

I suppose, this is your cluster?
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B2OZJSTAAOPFQ3VNWF24KLSOASGTOOW4/

To give some Proxmox related hints. I will assume a couple of things, please correct me if I am wrong.

I started to change size=1 and min_size=1. (i know, i know, i will never ever to that again!)

Please use size 3/2 for data protection. Even with 2/1, an PG could be in-flight and might not be written out to disk yet. Any subsequent failure can lead to data loss (especially in small clusters).

Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now:

Never run Corosync on a shared media (eg. Ceph+Corosync), always use at least one (better two) physical NIC ports, solely for Corosync. The reason for this is, that Corosync want a low and stable latency. Any other traffic will interfere.

My goal is to change that 3-Node Cluster with ceph storage to a Single-Node to get 2 free for a fresh install.

As my colleague suggested, first upgrade all nodes, then re-install one-by-one. The advantage is to only lose one node not two. Also Corosync switched from multicast to unicast with Corosync 3 and it allows to specify priorities for multiple links. This make handling the cluster communication easier.