[SOLVED] pveceph osd destroy is not cleaning device properly

[cmonty14]

Hi,
to remove an OSD I run this command:
root@ld5506:/var/lib/vz# pveceph osd destroy 2
destroy OSD osd.2
Remove osd.2 from the CRUSH map
Remove the osd.2 authentication key.
Remove OSD osd.2
--> Zapping: /dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be
--> Unmounting /var/lib/ceph/osd/ceph-2
Running command: /bin/umount -v /var/lib/ceph/osd/ceph-2
stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted
Running command: /sbin/wipefs --all /dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be
Running command: /bin/dd if=/dev/zero of=/dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be bs=1M count=10
stderr: 10+0 records in
10+0 records out
stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0233575 s, 449 MB/s
--> Zapping successful for OSD: 2

However, if I want to add the same device again and create an OSD, this fails because the device is not cleaned properly:
root@ld5506:~# pveceph osd create /dev/nvme0n1
device '/dev/nvme0n1' is already in use

I must delete manully the relevant LVM VG and LVM Logical volume before in order to create a new OSD:
root@ld5506:/var/lib/vz# vgremove --force ceph-638fd90e-6a71-477b-b7e5-f3006476328b
Logical volume "osd-block-9f73399c-d6a9-4253-b469-5df7b66589be" successfully removed
Volume group "ceph-638fd90e-6a71-477b-b7e5-f3006476328b" successfully removed

THX

 

[dcsapak]

use the 'cleanup' option

pveceph osd destroy ID --cleanup 1

see 'man pveceph'

 

[cmonty14]

Hm...
the manpage of pveceph shows
--cleanup <boolean> (default = 0)
If set, we remove partition table entries.

My understanding of this is not that LVM, means volume group and logical volume, will be removed.

Can you confirm that this option will remove volume group and logical volume?

 

[dcsapak]

Can you confirm that this option will remove volume group and logical volume?

yes it will remove the lvs/vgs/etc. the description text was not adapted yet

 

[dlasher]

use the 'cleanup' option

pveceph osd destroy ID --cleanup 1

see 'man pveceph'

Just wanted to be a big +1 for this command - I've been doing it the hard way any time a drive fails, and was pleasantly surprised to find it cleaned up all the pv/vg's correctly, even in the case of DB on NVME.

10/10 will use again.

 

[Dadsun]

If the wal device is on another disc, note that the lvm on the wal device isn't cleaned up.