Reinstall CEPH on Proxmox 6

[AngryAdm]

True. I meant a more improved version of the pveceph purge is available in Proxmox VE 6.2.

You improved it even more in version 7. So when you run pveceph purge on node #6 the configuration vanishes from all other nodes, essentielly destroying our ceph cluster.. I clap my hands at such excellent software behaviour!

Worst of all, after having done that, guess what... the problem that arose... is THIS THREAD all over again 1½ years later. When can we expect this to be fixed?

 

[t.lamprecht]

You improved it even more in version 7. So when you run pveceph purge on node #6 the configuration vanishes from all other nodes, essentielly destroying our ceph cluster.. I clap my hands at such excellent software behaviour!

You run a purge and it purged, wouldn't call that unexpected.. In general I'd recommend having backups also of configurations.

 

[DemiNe0]

In pve7, this is the script I was able to use to reinstall ceph without issue:

systemctl restart pvestatd
rm -rf /etc/systemd/system/ceph*
killall -9 ceph-mon ceph-mgr ceph-mds
rm -rf /etc/ceph /etc/pve/ceph.conf /etc/pve/priv/ceph* /var/lib/ceph
pveceph purge
systemctl restart pvestatd
apt purge ceph-mon ceph-osd ceph-mgr ceph-mds
systemctl restart pvestatd
rm /etc/init.d/ceph
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
dpkg-reconfigure ceph-base
dpkg-reconfigure ceph-mds
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
systemctl restart pvestatd
mkdir -p /etc/ceph
mkdir -p /var/lib/ceph/bootstrap-osd
mkdir -p /var/lib/ceph/mgr
mkdir -p /var/lib/ceph/mon
pveceph install
systemctl restart pvestatd
pveceph init
systemctl restart pvestatd

 

[m.v.k]

In pve7, this is the script I was able to use to reinstall ceph without issue:

systemctl restart pvestatd
rm -rf /etc/systemd/system/ceph*
killall -9 ceph-mon ceph-mgr ceph-mds
rm -rf /etc/ceph /etc/pve/ceph.conf /etc/pve/priv/ceph* /var/lib/ceph
pveceph purge
systemctl restart pvestatd
apt purge ceph-mon ceph-osd ceph-mgr ceph-mds
systemctl restart pvestatd
rm /etc/init.d/ceph
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
dpkg-reconfigure ceph-base
dpkg-reconfigure ceph-mds
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
systemctl restart pvestatd
mkdir -p /etc/ceph
mkdir -p /var/lib/ceph/bootstrap-osd
mkdir -p /var/lib/ceph/mgr
mkdir -p /var/lib/ceph/mon
pveceph install
systemctl restart pvestatd
pveceph init
systemctl restart pvestatd

this worked for me thx

 

[alain]

Hi DemiNeo,

Thanks also for your tutorial. I just used it for a new cluster installation. It was with Proxmox 8, so a more recent version, and your procedure worked perfectly. I could remove correctly Ceph, and reinstall it afrewards, in my case using the web interface ratehr than pveceph install.

it's a long time I setup a ceph cluster. I got confused by the documentation concerning Ceph setup for network. It mentions two networks, public network and cluster network, saying :
"Public Network: You can set up a dedicated network for Ceph....."
Cluster Network: As an optional step, you can go even further andseparate the OSD replication & heartbeat trafficas well..."

I had two networks configured, one accessible from the LAN, for Proxmox Corosync, and one, only internal; for storage, and indeed Ceph. First, I configured the public network as the first, corosync network, and the cluster network, with the second. It was indeed the correct setting. Then I had doubts, thinking storage has no need to have public network, and change the 'public network' for the "internal only storage.
This messed up the entire configuration, and had to reinstall. It is working now.

But perhaps the documentation should be clearer on this point.
https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

 

[skellar2006]

I fixed it for my situation.

My final solution to reinstall CEPH on all nodes:

rm -rf /etc/systemd/system/ceph*
killall -9 ceph-mon ceph-mgr ceph-mds
rm -rf /var/lib/ceph/mon/  /var/lib/ceph/mgr/  /var/lib/ceph/mds/
pveceph purge
apt purge ceph-mon ceph-osd ceph-mgr ceph-mds
rm /etc/init.d/ceph
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
dpkg-reconfigure ceph-base
dpkg-reconfigure ceph-mds
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
pveceph install

Thank you for posting this. It was helpful today! My 3 nodes have CEPH installed and working again. Thank you

 

[dewaldndende]

In pve7, this is the script I was able to use to reinstall ceph without issue:

systemctl restart pvestatd
rm -rf /etc/systemd/system/ceph*
killall -9 ceph-mon ceph-mgr ceph-mds
rm -rf /etc/ceph /etc/pve/ceph.conf /etc/pve/priv/ceph* /var/lib/ceph
pveceph purge
systemctl restart pvestatd
apt purge ceph-mon ceph-osd ceph-mgr ceph-mds
systemctl restart pvestatd
rm /etc/init.d/ceph
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
dpkg-reconfigure ceph-base
dpkg-reconfigure ceph-mds
dpkg-reconfigure ceph-common
dpkg-reconfigure ceph-fuse
for i in $(apt search ceph | grep installed | awk -F/ '{print $1}'); do apt reinstall $i; done
systemctl restart pvestatd
mkdir -p /etc/ceph
mkdir -p /var/lib/ceph/bootstrap-osd
mkdir -p /var/lib/ceph/mgr
mkdir -p /var/lib/ceph/mon
pveceph install
systemctl restart pvestatd
pveceph init
systemctl restart pvestatd

This worked ...until I created first mon , then no other nodes could see ceph cluster - all timeouts

 

[TheSmoker]

This worked ...until I created first mon , then no other nodes could see ceph cluster - all timeouts

I have the same issue on Proxmox 8.2. As soon as I create the first monitor, then I cannot create the others.