Reinstall/Reinit Ceph

[jehster]

Hi,

I've tried to get Ceph working on my proxmox cluster, but it failed. Many OSD haven't been created and actually I can't destroy existing pool and/or add OSD.

I wass wondering if there is a way to fully reinit ceph without reinstall proxmox

I have a 4 nodes cluster running Ceph 5.2.9.

Feel free to ask if more informations are needed.

regards
Jérôme

 

[jehster]

Hi,

I've tried to get Ceph working on my proxmox cluster, but it failed. Many OSD haven't been created and actually I can't destroy existing pool and/or add OSD.

I wass wondering if there is a way to fully reinit ceph without reinstall proxmox

I have a 4 nodes cluster running Ceph 5.2.9.

Feel free to ask if more informations are needed.

regards
Jérôme

Just adding some addtionnal informations, because I'm quite sure that one of you is going to tell it to me :

I already try 'pveceph purge' but it did not work (all ceph services were stopped)

 

[kaltsi]

Could you please share more details about your system?
Why is it failed?
Please share the "ceph status" response.

 

[jehster]

Sure,
Cluster is based on a Dell C6100
4 nodes with Proxmox 5.2.9, 96GB RAM, 2xL5639 by node.
1Gb network card on each that goes on the firewall (throught switch) and bonding that goes on storage network (I have another Ceph cluster, and Synology that share storage with proxmox cluster)

2 Pfsense (active/backup) that connect to internet

Last time I tried to destroy my pool, I did it using either GUI or CLI ... I had to stop task after many hours (many days !) runnning without doing anything.

It looks like I have no more configuration but I still see informations in GUI such as my last monitor (I managed to delete all other but last one seems alive)

Here's output of ceph status (that confirm that there is no more configuration files)

root@px01:/var/log/ceph# ceph status
2018-10-03 17:22:24.127251 7f85ffc0a700 -1 Errors while parsing config file!
2018-10-03 17:22:24.127258 7f85ffc0a700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-10-03 17:22:24.127259 7f85ffc0a700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-10-03 17:22:24.127259 7f85ffc0a700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)

 

[kaltsi]

Did you try reinstall the Ceph?

 

[jehster]

No I did not because the only way I've found was to remove package ceph but last time I'll try this I remember that it implies to remove packages such as pve-manager.

I've just tried and it seems not be true. It's quite late here so I won't do it right now. I'll do it tomorrow and give feedback here tomorrow.

thanks for helping

 

[spirit]

#pveceph purge

it shoud remove ceph cluster package without removing librbd (so pve-manager)

 

[jehster]

As said in my 2nd post, pveceph purge did not worked

After a good night, I found that I still had a ceph-osd process on one node. I killed the process.
Ran again pveceph purge and manage to recreate my cluster.

I managed to create OSD but got some error using web interface. So I used CLI to create osd

When I've created OSD, I never had error but some of them never get up. I had to to ceph-disk zap and recreate OSD

Now ceph is working