[SOLVED] Howto remove LUKs encryption from osd?

[jsterr]

This happend on osd creation on pve 7.1.2:

create OSD on /dev/nvme0n1 (bluestore)
wiping block device /dev/nvme0n1
/dev/nvme0n1: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme0n1: 8 bytes were erased at offset 0x5d268655e00 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme0n1: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/nvme0n1: calling ioctl to re-read partition table: Success
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.141035 s, 1.5 GB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 136ce71f-be23-4b8a-ae89-6432f36c9e81
Running command: /sbin/vgcreate --force --yes ceph-d40b53a1-b746-4554-8b90-cd58b7016f25 /dev/nvme0n1
 stdout: Physical volume "/dev/nvme0n1" successfully created.
 stdout: Volume group "ceph-d40b53a1-b746-4554-8b90-cd58b7016f25" successfully created
Running command: /sbin/lvcreate --yes -l 1526177 -n osd-block-136ce71f-be23-4b8a-ae89-6432f36c9e81 ceph-d40b53a1-b746-4554-8b90-cd58b7016f25
 stdout: Logical volume "osd-block-136ce71f-be23-4b8a-ae89-6432f36c9e81" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-d40b53a1-b746-4554-8b90-cd58b7016f25/osd-block-136ce71f-be23-4b8a-ae89-6432f36c9e81
Running command: /sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-d40b53a1-b746-4554-8b90-cd58b7016f25/osd-block-136ce71f-be23-4b8a-ae89-6432f36c9e81 wpAAFw-9T2t-UHL8-IhwE-Fo46-mbvt-XS7Vzp
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-4
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/mapper/wpAAFw-9T2t-UHL8-IhwE-Fo46-mbvt-XS7Vzp
Running command: /bin/chown -R ceph:ceph /dev/dm-1
Running command: /bin/ln -s /dev/mapper/wpAAFw-9T2t-UHL8-IhwE-Fo46-mbvt-XS7Vzp /var/lib/ceph/osd/ceph-4/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-4/activate.monmap
 stderr: 2022-03-01T13:17:23.712+0100 7f81a6338700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2022-03-01T13:17:23.712+0100 7f81a6338700 -1 AuthRegistry(0x7f81a005b2e8) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: got monmap epoch 3
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-4/keyring --create-keyring --name osd.4 --add-key AQDJDh5idJq9BRAAh1LjEhmCyXRegntPsZsyLA==
 stdout: creating /var/lib/ceph/osd/ceph-4/keyring
added entity osd.4 auth(key=AQDJDh5idJq9BRAAh1LjEhmCyXRegntPsZsyLA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid 136ce71f-be23-4b8a-ae89-6432f36c9e81 --setuser ceph --setgroup ceph
 stderr: 2022-03-01T13:17:23.888+0100 7f23aade7f00 -1 bluestore(/var/lib/ceph/osd/ceph-4/) _read_fsid unparsable uuid
 stderr: 2022-03-01T13:17:23.892+0100 7f23aade7f00 -1 bluefs _replay 0x0: stop: uuid 7d009bc0-9f12-d081-9dc0-cafb93403e09 != super.uuid 7a2f291f-5e7d-4bb6-ac83-58874c0659ad, block dump:
 stderr: 00000000  66 c9 43 a4 2c 58 7d 00  9b c0 9f 12 d0 81 9d c0  |f.C.,X}.........|
....
 stderr: 00001000
 stderr: 2022-03-01T13:17:24.420+0100 7f23aade7f00 -1 rocksdb: verify_sharding unable to list column families: NotFound:
 stderr: 2022-03-01T13:17:24.420+0100 7f23aade7f00 -1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db erroring opening db:
 stderr: 2022-03-01T13:17:24.960+0100 7f23aade7f00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
 stderr: 2022-03-01T13:17:24.960+0100 7f23aade7f00 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4/: (5) Input/output error[0m
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.4 --yes-i-really-mean-it
 stderr: 2022-03-01T13:17:25.016+0100 7f7c4e5c0700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2022-03-01T13:17:25.016+0100 7f7c4e5c0700 -1 AuthRegistry(0x7f7c4805b2e8) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: purged osd.4
-->  RuntimeError: Command failed with exit code 250: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid 136ce71f-be23-4b8a-ae89-6432f36c9e81 --setuser ceph --setgroup ceph
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid f971968e-a224-46fb-8c5c-49e01c12fabe --crush-device-class nvme --data /dev/nvme0n1 --dmcrypt' failed: exit code 1

I cant wipe the disk, and I cant remove the LuksKey with cryptsetup LuksRemoveKey because I need a password. How can I find the password?

Source: https://www.thegeekdiary.com/centos-rhel-how-to-delete-luks-encrypted-device/

How can I wipe the disk to recreate the osds?

root@proxmox02:~# ls /dev/mapper/cep*
/dev/mapper/ceph--d40b53a1--b746--4554--8b90--cd58b7016f25-osd--block--136ce71f--be23--4b8a--ae89--6432f36c9e81
root@proxmox02:~# cryptsetup remove /dev/mapper/ceph--d40b53a1--b746--4554--8b90--cd58b7016f25-osd--block--136ce71f--be23--4b8a--ae89--6432f36c9e81
Device /dev/mapper/ceph--d40b53a1--b746--4554--8b90--cd58b7016f25-osd--block--136ce71f--be23--4b8a--ae89--6432f36c9e81 is still in use.

 

[jsterr]

I rebooted the server (which removed something somehow)) Then the following commands worked:

cryptsetup remove /dev/mapper/ceph--d40b53a1--b746--4554--8b90--cd58b7016f25-osd--block--136ce71f--be23--4b8a--ae89--6432f36c9e81
wipefs -a /dev/nvme0n1
sgdisk -Z /dev/nvme0n1

and then wiped and gpt formated the disks via gui

Recreation of osd worked fine without any problem.