[SOLVED] how to purge the lefted ceph monitor and ceph osds of a node which has been removed from a cluster before

[johnhk]

there is a four nodes pve cluster:

pve-s,pve1,pve2,pve4

After powering off the node pve1, I remove it from the cluster.

$ pvecm delnode pve1

but the related ceph monitor and ceph osd can't be destroyed any more.

how can I fix it？

 

[fiona]

Hi,
since the node doesn't exist anymore, I'm not sure it's even possible via Proxmox VE's API. Please try to follow the steps described in the Ceph documentation here and here.

 

[kenneth104]

you can test del the related information in /etc/pve/ceph.conf

 

[johnhk]

you can test del the related information in /etc/pve/ceph.con

I tried comment the related parts in ceph.conf

[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 10.0.0.32/24
         fsid = 8ba5c7b3-5f27-4121-9fe6-49edc01768bd
         mon_allow_pool_delete = true
#        mon_host = 10.0.0.32 10.0.0.30 10.0.0.31 10.0.0.34
         mon_host = 10.0.0.32 10.0.0.31 10.0.0.34
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_pool_default_min_size = 2
         osd_pool_default_size = 3
         public_network = 10.0.0.32/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.pve-s]
         host = pve-s
         mds_standby_for_name = pve

#[mds.pve1]
#        host = pve1
#        mds_standby_for_name = pve

[mds.pve2]
         host = pve2
         mds_standby_for_name = pve

[mon.pve-s]
         public_addr = 10.0.0.32

#[mon.pve1]
#        public_addr = 10.0.0.30

[mon.pve2]
         public_addr = 10.0.0.31

[mon.pve4]
         public_addr = 10.0.0.34

then , i restart the ceph

systemctl restart ceph.target

but there is no effect at all. the same as before

 

[johnhk]

Hi,
since the node doesn't exist anymore, I'm not sure it's even possible via Proxmox VE's API. Please try to follow the steps described in the Ceph documentation here and here.

thanks for your suggestion, I fixed it by:

remove the monitor

root@pve-s:/etc/pve# ceph mon dump
epoch 4
fsid 8ba5c7b3-5f27-4121-9fe6-49edc01768bd
last_changed 2022-08-26T11:07:52.447674+0800
created 2022-08-15T15:42:49.156532+0800
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:10.0.0.32:3300/0,v1:10.0.0.32:6789/0] mon.pve-s
1: [v2:10.0.0.30:3300/0,v1:10.0.0.30:6789/0] mon.pve1
2: [v2:10.0.0.31:3300/0,v1:10.0.0.31:6789/0] mon.pve2
3: [v2:10.0.0.34:3300/0,v1:10.0.0.34:6789/0] mon.pve4
dumped monmap epoch 4
root@pve-s:/etc/pve# ceph mon remove pve1
removing mon.pve1 at [v2:10.0.0.30:3300/0,v1:10.0.0.30:6789/0], there will be 3 monitors
root@pve-s:/etc/pve# ceph mon dump
epoch 5
fsid 8ba5c7b3-5f27-4121-9fe6-49edc01768bd
last_changed 2022-08-26T17:28:08.588612+0800
created 2022-08-15T15:42:49.156532+0800
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:10.0.0.32:3300/0,v1:10.0.0.32:6789/0] mon.pve-s
1: [v2:10.0.0.31:3300/0,v1:10.0.0.31:6789/0] mon.pve2
2: [v2:10.0.0.34:3300/0,v1:10.0.0.34:6789/0] mon.pve4
dumped monmap epoch 5

removes the OSD&remove the host bucket

root@pve-s:/etc/pve# ceph osd purge 0
purged osd.0
ceph osd crush remove {name}
root@pve-s:/etc/pve# ceph osd crush remove pve1
removed item id -3 name 'pve1' from crush map