Reinstall/Reinit Ceph

jehster

Renowned Member
Jan 19, 2016
8
0
66
47
Hi,

I've tried to get Ceph working on my proxmox cluster, but it failed. Many OSD haven't been created and actually I can't destroy existing pool and/or add OSD.

I wass wondering if there is a way to fully reinit ceph without reinstall proxmox

I have a 4 nodes cluster running Ceph 5.2.9.

Feel free to ask if more informations are needed.

regards
Jérôme
 
Hi,

I've tried to get Ceph working on my proxmox cluster, but it failed. Many OSD haven't been created and actually I can't destroy existing pool and/or add OSD.

I wass wondering if there is a way to fully reinit ceph without reinstall proxmox

I have a 4 nodes cluster running Ceph 5.2.9.

Feel free to ask if more informations are needed.

regards
Jérôme
Just adding some addtionnal informations, because I'm quite sure that one of you is going to tell it to me :

I already try 'pveceph purge' but it did not work (all ceph services were stopped)
 
Sure,
Cluster is based on a Dell C6100
4 nodes with Proxmox 5.2.9, 96GB RAM, 2xL5639 by node.
1Gb network card on each that goes on the firewall (throught switch) and bonding that goes on storage network (I have another Ceph cluster, and Synology that share storage with proxmox cluster)

2 Pfsense (active/backup) that connect to internet

Last time I tried to destroy my pool, I did it using either GUI or CLI ... I had to stop task after many hours (many days !) runnning without doing anything.

It looks like I have no more configuration but I still see informations in GUI such as my last monitor (I managed to delete all other but last one seems alive)

Here's output of ceph status (that confirm that there is no more configuration files)

Code:
root@px01:/var/log/ceph# ceph status
2018-10-03 17:22:24.127251 7f85ffc0a700 -1 Errors while parsing config file!
2018-10-03 17:22:24.127258 7f85ffc0a700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-10-03 17:22:24.127259 7f85ffc0a700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-10-03 17:22:24.127259 7f85ffc0a700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
 
No I did not because the only way I've found was to remove package ceph but last time I'll try this I remember that it implies to remove packages such as pve-manager.

I've just tried and it seems not be true. It's quite late here so I won't do it right now. I'll do it tomorrow and give feedback here tomorrow.

thanks for helping
 
As said in my 2nd post, pveceph purge did not worked ;)

After a good night, I found that I still had a ceph-osd process on one node. I killed the process.
Ran again pveceph purge and manage to recreate my cluster.

I managed to create OSD but got some error using web interface. So I used CLI to create osd

When I've created OSD, I never had error but some of them never get up. I had to to ceph-disk zap and recreate OSD

Now ceph is working
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!