Hi there!
I had to do the cephx keyring migration by hand since I have a Ceph cluster member which is not part of my PVE cluster. I followed the upstream guide from ceph.com and had a healthy cluster at some point. That is after I rebooted every machine that had locked up because I rotated the admin key for the storage too late.
However, with every host reboot I perform now, the keyring files of the OSDs are empty. I didn't even know that they were recreated on every boot. This is the content:
Any hints how I can tell
Regards
Marco
I had to do the cephx keyring migration by hand since I have a Ceph cluster member which is not part of my PVE cluster. I followed the upstream guide from ceph.com and had a healthy cluster at some point. That is after I rebooted every machine that had locked up because I rotated the admin key for the storage too late.
However, with every host reboot I perform now, the keyring files of the OSDs are empty. I didn't even know that they were recreated on every boot. This is the content:
Code:
[osd.X]
key = keyring/var/log/ceph/ceph-volume.log shows something suspicious:
Code:
[2026-10-02 20:05:10,890][ceph_volume.main][INFO ] Running command: ceph-volume lvm trigger 6-beaa37f9-041c-4fed-bae7-d6127b66adbf
[2026-10-02 20:05:10,892][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2026-10-02 20:05:10,911][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2026-10-02 20:05:10,931][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is
[2026-10-02 20:05:10,931][ceph_volume.process][INFO ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S tags={ceph.osd_id=6,ceph.osd_fsid=beaa37f9-041c-4fed-bae7-d6127b66adbf} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2026-10-02 20:05:10,977][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 174, in main
terminal.dispatch(self.mapper, subcommand_args)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 47, in main
terminal.dispatch(self.mapper, self.argv)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/trigger.py", line 73, in main
Activate(['--auto-detect-objectstore', osd_id, osd_uuid]).main()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/activate.py", line 105, in main
self.objectstore.activate()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3/dist-packages/ceph_volume/objectstore/lvmbluestore.py", line 487, in activate
raise RuntimeError('could not find osd.%s with osd_fsid %s' %
(osd_id, osd_fsid))
RuntimeError: could not find osd.6 with osd_fsid beaa37f9-041c-4fed-bae7-d6127b66adbfAny hints how I can tell
ceph-volume what the correct fsid for each volume is? Why did it even change?Regards
Marco