reinstall ceph after pveceph purge

[Luukth]

Hi,

After I installed proxmox I decided to tinker around with ceph. Some things didn't work out and I removed ceph from the proxmox node. After stopping all the ceph services I removed it with 'pveceph purge'. That worked!

Now when I tried to reconfigure ceph I keep getting this error "Could not connect to ceph cluster despite configured monitors (500)". I Couldn't find anything on the internet that fixes the problem. In the process I might even have destroyed more... oops. Is there some magic command I can run the wipe ceph from the proxmox node and reinstall it. That might be the best option considering the damage I have already done. There is nothing important on this proxmox node, I just started out and I'm trying to learn.

● ceph-mon@proxmox.service - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/ceph-mon@.service.d
           └─ceph-after-pve-cluster.conf
   Active: failed (Result: exit-code) since Sat 2019-08-03 06:15:35 CDT; 9min ago
  Process: 1174 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id proxmox --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Main PID: 1174 (code=exited, status=1/FAILURE)

Aug 03 06:15:35 proxmox systemd[1]: ceph-mon@proxmox.service: Service RestartSec=10s expired, scheduling restart.
Aug 03 06:15:35 proxmox systemd[1]: ceph-mon@proxmox.service: Scheduled restart job, restart counter is at 5.
Aug 03 06:15:35 proxmox systemd[1]: Stopped Ceph cluster monitor daemon.
Aug 03 06:15:35 proxmox systemd[1]: ceph-mon@proxmox.service: Start request repeated too quickly.
Aug 03 06:15:35 proxmox systemd[1]: ceph-mon@proxmox.service: Failed with result 'exit-code'.
Aug 03 06:15:35 proxmox systemd[1]: Failed to start Ceph cluster monitor daemon.

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster network = 192.168.1.168/24
     fsid = b406262f-620c-4216-a912-a11b03023969
     mon_allow_pool_delete = true
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public network = 192.168.1.168/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

 

[Alwin]

You need to remove all service files and the config + any leftover /var/lib/ceph/ directory. To remove ceph, you would first destroy all the services (eg. pveceph mon/mgr/osd destroy) and then run the purge.

 

[Luukth]

You need to remove all service files and the config + any leftover /var/lib/ceph/ directory. To remove ceph, you would first destroy all the services (eg. pveceph mon/mgr/osd destroy) and then run the purge.

Thanks for your reply, I have already decided to reinstall Ceph. Let's hope people who see this thread can get it working again with your recommendations.

 

[oz1cw7yymn]

@Alwin While I have also reinstalled pve from scratch again, the steps I described in my thread (https://forum.proxmox.com/threads/ceph-reinstallation-issues.56580/) would be the same I thought, but it didn't work for me. Was I missing any particular step?

 

[Alwin]

@Alwin While I have also reinstalled pve from scratch again, the steps I described in my thread (https://forum.proxmox.com/threads/ceph-reinstallation-issues.56580/) would be the same I thought, but it didn't work for me. Was I missing any particular step?

First you need to remove the services, through pveceph or the ceph tooling. Then run the pveceph purge, or delete the config (and link) by hand and remove the /var/lib/ceph folder. If I read the other post correctly, then you ran the purge first.

 

[oz1cw7yymn]

Thanks for the reply. I'm not sure about the order anymore, I tried many times, but I believe I only ran purge after my ceph installation was already borked and it was no longer possible to use the tools to stop/remove the services.

Basically, I was trying to clean out ceph after it was no longer functioning in order to reinstall from scratch. I tried by removing all folders, config files and packages related to the ceph server installation, while keeping the files needed for pve as a ceph client. Is that not possible and a pve installation is the only solution?

 

[Alwin]

Basically, I was trying to clean out ceph after it was no longer functioning in order to reinstall from scratch. I tried by removing all folders, config files and packages related to the ceph server installation, while keeping the files needed for pve as a ceph client. Is that not possible and a pve installation is the only solution?

This is certainly possible. The packages can stay installed, some of them are needed if you want to have the Nautilus client.

 

[Ian Macdonald]

I have the same issue, going to reinstall Proxmox tried the recommendations here, something is missing, Ceph seem very fragile when used with Proxmox 6.