Hi
I have found some documentation of the case that I am facing, but wanted to run it by some of experts here first
We currently have a 5-node PVE cluster where 4 nodes are running a Ceph cluster and with Proxmox HA configured for all VMs on these nodes. Each node has two NVMe drives each configured as an OSD in Ceph.
The original plan was to have 3 nodes run with 4 NVMe drives each as well as some more powerful CPUs. But due to the current situation with limited access to the subcomponents for the required hardware components. these servers were on a backorder and we had to order some different servers to increase cluster capacity.
Now our supplier has finally been able to get hold of these original servers ordered and we need to plan on putting these into the cluster.
From what I have understood, we need to add the new servers to the Proxmox cluster and also to the Ceph cluster and create OSDs. Once that is done, we need to get the 4 current nodes removed as they are needed elsewhere. For this we need to mark and OSD "out" one by one and allow Ceph to rebalance the data on the remaining OSDs. Once I have marked all OSDs of a given server "out", do I then need to remove them from the cluster? Of course MON, MGR and MDS needs to be stopped/removed as well, right?
Any other steps required to succesfully remove the nodes from the ceph cluster?
Additionally, will any type of manual configuration be required when going from 4 nodes to 3 nodes?
I have found some documentation of the case that I am facing, but wanted to run it by some of experts here first
We currently have a 5-node PVE cluster where 4 nodes are running a Ceph cluster and with Proxmox HA configured for all VMs on these nodes. Each node has two NVMe drives each configured as an OSD in Ceph.
The original plan was to have 3 nodes run with 4 NVMe drives each as well as some more powerful CPUs. But due to the current situation with limited access to the subcomponents for the required hardware components. these servers were on a backorder and we had to order some different servers to increase cluster capacity.
Now our supplier has finally been able to get hold of these original servers ordered and we need to plan on putting these into the cluster.
From what I have understood, we need to add the new servers to the Proxmox cluster and also to the Ceph cluster and create OSDs. Once that is done, we need to get the 4 current nodes removed as they are needed elsewhere. For this we need to mark and OSD "out" one by one and allow Ceph to rebalance the data on the remaining OSDs. Once I have marked all OSDs of a given server "out", do I then need to remove them from the cluster? Of course MON, MGR and MDS needs to be stopped/removed as well, right?
Any other steps required to succesfully remove the nodes from the ceph cluster?
Additionally, will any type of manual configuration be required when going from 4 nodes to 3 nodes?