Proxmox VE 6 CEPH renew failure

Jul 25, 2019
4
1
3
37
We have a cluster of four almost identical servers:
- CPU E5-2620v4@2.10GHz (2x16core)
- RAM 128, 128, 256, 256 GB
- 4x NIC 10Gb, 2x1Gb
- 2x SSD 400GB (PVE)
- 6x SSD 1TB (VM's on CEPH)
- 8x HDD 1TB (backup, misc)

The first server has PVE5 (will get a fresh PVE6 soon), the others have PVE6 installed.
This cluster is in full production, but undergoing drastic changes.

Initially the servers had Ubuntu, later 1-by-one PVE was installed.
All servers are at this moment stand-alone, no shared storage except /etc/pve
Everything is working well, except for CEPH, we just want a fresh/new start with CEPH.

We tried several guides on the forum and elsewhere, but are stuck and need help.
It appears not possible to remove the old CEPH setup, in particulair a mon.

All we need is instructions to be able to restore all servers in a fresh state for CEPH .
The CEPH osd's can be wiped, all VM's are on HDD-storage (ZFS), including vzdump back-ups.

We applied the following steps to refresh CEPH, but it appears not entirely good:


## stop all ceph-services
systemctl stop ceph-mon.target
systemctl stop ceph-mgr.target
systemctl stop ceph-mds.target
systemctl stop ceph-osd.target

## avoid that they're being restarted by systemd the next boot (the low level way)
rm -rf /etc/systemd/system/ceph*

## be really sure they're stopped:
killall -9 ceph-mon ceph-mgr ceph-mds

## remove CEPH data
rm -rf /var/lib/ceph/mon/ /var/lib/ceph/mgr/ /var/lib/ceph/mds/

## PVE CLI to remove CEPH config
pveceph purge

## wipe all OSD's ussing the following ($DISK initialised in a loop):
#ceph-volume lvm zap /dev/$DISK --destroy
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!