[SOLVED] pveceph osd destroy is not cleaning device properly

cmonty14

Well-Known Member
Mar 4, 2014
343
5
58
Hi,
to remove an OSD I run this command:
root@ld5506:/var/lib/vz# pveceph osd destroy 2
destroy OSD osd.2
Remove osd.2 from the CRUSH map
Remove the osd.2 authentication key.
Remove OSD osd.2
--> Zapping: /dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be
--> Unmounting /var/lib/ceph/osd/ceph-2
Running command: /bin/umount -v /var/lib/ceph/osd/ceph-2
stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted
Running command: /sbin/wipefs --all /dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be
Running command: /bin/dd if=/dev/zero of=/dev/ceph-638fd90e-6a71-477b-b7e5-f3006476328b/osd-block-9f73399c-d6a9-4253-b469-5df7b66589be bs=1M count=10
stderr: 10+0 records in
10+0 records out
stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0233575 s, 449 MB/s
--> Zapping successful for OSD: 2


However, if I want to add the same device again and create an OSD, this fails because the device is not cleaned properly:
root@ld5506:~# pveceph osd create /dev/nvme0n1
device '/dev/nvme0n1' is already in use


I must delete manully the relevant LVM VG and LVM Logical volume before in order to create a new OSD:
root@ld5506:/var/lib/vz# vgremove --force ceph-638fd90e-6a71-477b-b7e5-f3006476328b
Logical volume "osd-block-9f73399c-d6a9-4253-b469-5df7b66589be" successfully removed
Volume group "ceph-638fd90e-6a71-477b-b7e5-f3006476328b" successfully removed


THX
 
use the 'cleanup' option
Code:
pveceph osd destroy ID --cleanup 1

see 'man pveceph'
 
  • Like
Reactions: cmonty14
Hm...
the manpage of pveceph shows
--cleanup <boolean> (default = 0)
If set, we remove partition table entries.


My understanding of this is not that LVM, means volume group and logical volume, will be removed.

Can you confirm that this option will remove volume group and logical volume?
 
Can you confirm that this option will remove volume group and logical volume?
yes it will remove the lvs/vgs/etc. the description text was not adapted yet
 
use the 'cleanup' option
Code:
pveceph osd destroy ID --cleanup 1

see 'man pveceph'

Just wanted to be a big +1 for this command - I've been doing it the hard way any time a drive fails, and was pleasantly surprised to find it cleaned up all the pv/vg's correctly, even in the case of DB on NVME.

10/10 will use again. :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!