[SOLVED] Ceph create OSD on unused disk failed, disk marked as used now

[justjosh]

Tried to create OSD on unused disk, process failed with "can't open exclusively".

trying to acquire lock...
OK
create OSD on /dev/sdd (bluestore)
wipe disk/partition: /dev/sdd
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.489536 s, 428 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 774221ff-1edf-4931-a804-1b4c606bc872
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-d301ac04-78c3-4dc9-a2c6-5029262d3db4 /dev/sdd
stderr: Can't open /dev/sdd exclusively. Mounted filesystem?
stderr: Can't open /dev/sdd exclusively. Mounted filesystem?
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.15 --yes-i-really-mean-it
stderr: 2020-01-02 23:29:33.816 7ff205fa5700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2020-01-02 23:29:33.816 7ff205fa5700 -1 AuthRegistry(0x7ff200080208) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: purged osd.15
--> RuntimeError: command returned non-zero exit status: 5
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid f3ad2f92-8e4b-400e-9309-0c58b824e567 --data /dev/sdd' failed: exit code 1

Now disk is marked as used by "Device Mapper". fdisk -l lists 3 OSDs (2 successfully created + the failed one).
Can't zap it to clear it.

ceph-volume lvm zap /dev/sdd --destroy
--> Zapping: /dev/sdd
Running command: /usr/sbin/wipefs --all /dev/sdd
stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> RuntimeError: command returned non-zero exit status: 1

 

[sg90]

Have you tried to restart the server? Sounds like a process still has a lock on /dev/sdd