We have a cluster of four almost identical servers:
- CPU E5-2620v4@2.10GHz (2x16core)
- RAM 128, 128, 256, 256 GB
- 4x NIC 10Gb, 2x1Gb
- 2x SSD 400GB (PVE)
- 6x SSD 1TB (VM's on CEPH)
- 8x HDD 1TB (backup, misc)
The first server has PVE5 (will get a fresh PVE6 soon), the others have PVE6 installed.
This cluster is in full production, but undergoing drastic changes.
Initially the servers had Ubuntu, later 1-by-one PVE was installed.
All servers are at this moment stand-alone, no shared storage except /etc/pve
Everything is working well, except for CEPH, we just want a fresh/new start with CEPH.
We tried several guides on the forum and elsewhere, but are stuck and need help.
It appears not possible to remove the old CEPH setup, in particulair a mon.
All we need is instructions to be able to restore all servers in a fresh state for CEPH .
The CEPH osd's can be wiped, all VM's are on HDD-storage (ZFS), including vzdump back-ups.
We applied the following steps to refresh CEPH, but it appears not entirely good:
## stop all ceph-services
systemctl stop ceph-mon.target
systemctl stop ceph-mgr.target
systemctl stop ceph-mds.target
systemctl stop ceph-osd.target
## avoid that they're being restarted by systemd the next boot (the low level way)
rm -rf /etc/systemd/system/ceph*
## be really sure they're stopped:
killall -9 ceph-mon ceph-mgr ceph-mds
## remove CEPH data
rm -rf /var/lib/ceph/mon/ /var/lib/ceph/mgr/ /var/lib/ceph/mds/
## PVE CLI to remove CEPH config
pveceph purge
## wipe all OSD's ussing the following ($DISK initialised in a loop):
#ceph-volume lvm zap /dev/$DISK --destroy
- CPU E5-2620v4@2.10GHz (2x16core)
- RAM 128, 128, 256, 256 GB
- 4x NIC 10Gb, 2x1Gb
- 2x SSD 400GB (PVE)
- 6x SSD 1TB (VM's on CEPH)
- 8x HDD 1TB (backup, misc)
The first server has PVE5 (will get a fresh PVE6 soon), the others have PVE6 installed.
This cluster is in full production, but undergoing drastic changes.
Initially the servers had Ubuntu, later 1-by-one PVE was installed.
All servers are at this moment stand-alone, no shared storage except /etc/pve
Everything is working well, except for CEPH, we just want a fresh/new start with CEPH.
We tried several guides on the forum and elsewhere, but are stuck and need help.
It appears not possible to remove the old CEPH setup, in particulair a mon.
All we need is instructions to be able to restore all servers in a fresh state for CEPH .
The CEPH osd's can be wiped, all VM's are on HDD-storage (ZFS), including vzdump back-ups.
We applied the following steps to refresh CEPH, but it appears not entirely good:
## stop all ceph-services
systemctl stop ceph-mon.target
systemctl stop ceph-mgr.target
systemctl stop ceph-mds.target
systemctl stop ceph-osd.target
## avoid that they're being restarted by systemd the next boot (the low level way)
rm -rf /etc/systemd/system/ceph*
## be really sure they're stopped:
killall -9 ceph-mon ceph-mgr ceph-mds
## remove CEPH data
rm -rf /var/lib/ceph/mon/ /var/lib/ceph/mgr/ /var/lib/ceph/mds/
## PVE CLI to remove CEPH config
pveceph purge
## wipe all OSD's ussing the following ($DISK initialised in a loop):
#ceph-volume lvm zap /dev/$DISK --destroy
Last edited:
