Ceph purge leave some traces behind, can't reconfigure cluster

[SilverXXX]

Hi everyone
I'm having some troubles, i'm trying to recreate ceph cluster in a simple homelab installation, but after pveceph purge and purging packages, when i try to reconfigure the cluster i get "Could not connect to ceph cluster despite configured monitors (500)".
Does purge leave something behind on purpose that must be manually removed?

Thanks

Small edit: the pool i created the first time still shows up in web gui

 

[udo]

Hi,
is the ceph-pool still declared in /etc/pve/storage.cfg?

any ceph-files left?

find /etc -ls | grep ceph

* some files are normal:

find /etc -ls | grep ceph
   190028      6 -rw-r--r--   1 root     root          159 Jan 30  2019 /etc/default/ceph
    10097      1 drwxr-xr-x   2 root     root            3 Sep 19 23:04 /etc/ceph
    10098      1 -rw-r--r--   1 root     root           92 Nov 19  2018 /etc/ceph/rbdmap
    15743      1 lrwxrwxrwx   1 root     root           31 Jul 30 23:55 /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target
    14524      1 lrwxrwxrwx   1 root     root           36 Jul 30 23:55 /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target
    14525      1 drwxr-xr-x   2 root     root            3 Jul 30 23:55 /etc/systemd/system/ceph.target.wants
    14526      1 lrwxrwxrwx   1 root     root           36 Jul 30 23:55 /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target
   190030      6 -rw-r--r--   1 root     root          237 Jan 30  2019 /etc/logrotate.d/ceph-common

Udo

 

[SilverXXX]

Yes, it was still in storage.cfg.

There are still many files even with purge:

  5505502      4 -rw-r--r--   1 root     root           49 Sep 19 13:49 /etc/sysctl.d/30-ceph-osd.conf
  5505490      4 -rw-r--r--   1 root     root         1488 Sep 19 13:45 /etc/bash_completion.d/ceph
  5505501      4 -r--r-----   1 root     root          197 Sep 19 13:49 /etc/sudoers.d/ceph-osd-smartctl
  5505517      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target
  5505525      0 lrwxrwxrwx   1 root     root           31 Aug 24 14:04 /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target
  5505515      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target
  5505513      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target
  5505511      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target
  5506326      0 lrwxrwxrwx   1 root     root           36 Aug 24 14:04 /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target
  5505521      4 drwxr-xr-x   2 root     root         4096 Sep 28 13:30 /etc/systemd/system/ceph-mon.target.wants
  5505522      0 lrwxrwxrwx   1 root     root           37 Sep 28 13:30 /etc/systemd/system/ceph-mon.target.wants/ceph-mon@pvemain.service -> /lib/systemd/system/ceph-mon@.service
  5506327      4 drwxr-xr-x   2 root     root         4096 Sep 28 13:26 /etc/systemd/system/ceph.target.wants
  5505518      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target
  5505516      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target
  5506328      0 lrwxrwxrwx   1 root     root           36 Aug 24 14:04 /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target
  5505510      0 lrwxrwxrwx   1 root     root           38 Sep 28 13:26 /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service
  5505514      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target
  5505512      0 lrwxrwxrwx   1 root     root           35 Sep 28 13:26 /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target
  5505681      4 drwxr-xr-x   2 root     root         4096 Sep 28 14:53 /etc/systemd/system/ceph-mds.target.wants
  5505973      0 lrwxrwxrwx   1 root     root           37 Sep 28 14:53 /etc/systemd/system/ceph-mds.target.wants/ceph-mds@pvemain.service -> /lib/systemd/system/ceph-mds@.service
  5505523      4 drwxr-xr-x   2 root     root         4096 Sep 28 13:30 /etc/systemd/system/ceph-mgr.target.wants
  5505524      0 lrwxrwxrwx   1 root     root           37 Sep 28 13:30 /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@pvemain.service -> /lib/systemd/system/ceph-mgr@.service
  5505488      4 drwxr-xr-x   2 ceph     ceph         4096 Sep 30 00:23 /etc/ceph
  5505519      4 -rw-------   1 ceph     ceph          151 Sep 29 19:01 /etc/ceph/ceph.client.admin.keyring
  5505496      4 -rw-r--r--   1 root     root          308 Sep 19 13:49 /etc/logrotate.d/ceph-common
  5505484      4 -rw-r--r--   1 root     root           65 Sep 29 19:00 /etc/apt/sources.list.d/ceph.list
  5505679      4 -rw-r--r--   1 root     root          159 Jan 30  2019 /etc/default/ceph

And for some operation i get :
unable to get monitor info from DNS SRV with service name: ceph-mon

 

[SilverXXX]

i can't find anything, the only thing i found is that it seems a message from one of ceph binaries and not from a proxmox script.
Even after reinstalling some packages it still remains the same

 

[Alwin]

unable to get monitor info from DNS SRV with service name: ceph-mon

What are you trying to run?

 

[G0ldmember]

I have the same problem - I wanted to uninstall ceph / delete ceph cluster and re-setup with a new config.

I tried pveceph purge, apt purge ceph, r- rf /var/lib/ceph, pveceph install

All that seemed to work, but when I initialise ceph with pveceph init --network x.x.x.x and then call pveceph createmon I also get

unable to get monitor info from DNS SRV with service name: ceph-mon
Could not connect to ceph cluster despite configured monitors

Is there a way to get rid of the old ceph stuff without reinstalling the system?

 

[Jeff Wadsworth]

You could try the steps here, but no guarantee.
https://forum.proxmox.com/threads/reinstall-ceph-on-proxmox-6.57691/

Also, for recovering the entire proxmox server to an earlier stable state:
https://forum.proxmox.com/threads/using-zfs-snapshots-on-rpool-root-pve-1.27530/