[PVE+CEPH] creating new OSD fails :: invalid cephx secret :: ceph-volume lvm create exit code 1

[aljaxus]

I have a 6-node [pve2 ... pve9] PVE cluster [base versions: pvemgr 9.1.4, ceph 19.2.3] and I have started updating the nodes. After pve6 node upgrade two small OSDs failed to start back up, I destroyed them [as they would have been removed soon anyway]. I then upgraded pve9 [to pve-manager/9.2.18 and ceph/19.2.6-pve4] and tried to add a new OSD - this issue starts here:


Creating a new OSD fails with the following output;

create OSD on /dev/sdf (bluestore)
wiping block device /dev/sdf
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.03505 s, 203 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 3bd22560-e8be-47b0-8964-25195d4bfcce
 stderr: Error EINVAL: invalid cephx secret.
-->  RuntimeError: Unable to create a new OSD id

TASK ERROR: command 'ceph-volume lvm create --data /dev/sdf --dmcrypt' failed: exit code 1

Creating via pveceph does [as expected] the same thing;

root@pve9:~# pveceph osd create /dev/sdf --encrypted=1 --crush-device-class=hdd
create OSD on /dev/sdf (bluestore)
wiping block device /dev/sdf
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.03561 s, 203 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b1d21242-ddca-4085-aba7-e06863249b7c
 stderr: Error EINVAL: invalid cephx secret.
-->  RuntimeError: Unable to create a new OSD id
command 'ceph-volume lvm create --crush-device-class hdd --data /dev/sdf --dmcrypt' failed: exit code 1

BUT, running the following commands manually succeeds;

root@pve9:~# /usr/bin/ceph-authtool --gen-print-key | /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ca2ee10b-08cb-4548-9e7c-2d722e872c00
15

---


I am unable to find ANY information about this issue that would be relevant. Everyone just "wipes ceph and re-initiates it" which is not a solution for me at all.

 

[j.theisen]

Hi @aljaxus

thanks for posting on the forum!

Did you already do the migration / rollover of the Ceph cluster keys following this docs section [1]?
What is the current Health state of the cluster?

Since you upgraded to Ceph 19.2.6 the older AES encryption is deprecated and a new key format is introduced.
It is strongly recommended to rotate the keys at your earliest convenience.

I also just tried to destroy and recreate an OSD with the new version before rolling over and that also worked.

Best regards
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_migration

 

[aljaxus]

Hi, I have decided to just migrate everything to a fresh cluster by moving nodes one-by one, and migrate VMs via Datacenter manager.
This cluster has been alive, upgraded, with nodes added, removed since early PVE7 - so it's high noon to start from scratch.

After I migrate all the VMs, I will keep 3 nodes in the old cluster with some OSDs so I can follow the upgrade guide, possibly fix the issue and report back.
Expect a follow up in a week.