Ceph removed per instructions - one node has errors

[tomfowler]

Hello,

I experimented with a four node ceph cluster and eventually decided it was not a good fit for me. I followed the instructions on the Proxmox support site and removed it from all the nodes. I have one node that continues to have errors posted to the logs. It started with lvm errors, which I researched and removed via the systemd disable command, but I am still left with crash logs every few seconds.

ceph crash ls:

Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

GUI error message:

Nov 21 09:30:07 darkbramble ceph-crash[832]: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create

I checked apt-cache and there are quite a few ceph packages installed. When I tested an apt remove ceph-common command, I didn't like some of the packages it wanted to remove (pve-manager, pve-ha-manager, pve-qemu-kvm...)

The ceph repository is disabled in the GUI.

Thanks for any help you can provide.

 

[sb-jw]

Check this: https://forum.proxmox.com/threads/removing-ceph-completely.62818/#post-604868

Can you share the history with us? Then we can also look at what you have already tried and what still needs to be done.

 

[tomfowler]

Thanks for your willingness to help!

This was a four node hyperconverged proxmox homelab. Using identical M900 Lenovo tiny units with 1TB SSD's on each. Ceph worked quite well and was able to do everything I wanted it to do. After a month or so, I found that I could do what I was really interested in doing with ZFS and HA. I can stand a few minutes of downtime if the node goes down while it brings up the VM on another node. So, I decided to remove the Ceph storage and followed the instructions here provided by a staff member:

Go to a nodes ceph tab and destroy first all OSDs, then all Managers and Monitors but not the last one.
On the last one just stop those two, then go to the command line and execute

pveceph purge

to remove the ceph packages for each node.

After the ceph storage was removed, I downsized the proxmox cluster to three nodes. Upon reviewing the logs, I'm seeing the above error messages on just one node. This wasn't even the last one I ran the purge command from. Not sure what happened to it.

The instructions you provided in the link above seems to have worked. No further messages in the syslog. I'll continue to monitor for the next few days. Thank you!

-Tom