osd.X carries no 'osd_key'

[lucacosmo]

Hi everyone,

I upgraded from Proxmox VE 8 to 9 without being aware that this release came with breaking changes around Ceph (I completely missed the CephX key migration requirement). I've since tried searching the docs and asking an LLM for help, but I haven't found a working solution.

My cluster is currently in HEALTH_WARN with the following checks:

8 auth client entities with insecure key types
Monitors are configured to allow auth using insecure key types
Monitors are configured to allow creation of insecure key types
4 rotating auth service keys using insecure key types
24 auth service entities with insecure key types
Monitors are configured to issue insecure service tickets
3 OSD(s) experiencing slow operations in BlueStore

Ceph Version : 19.2.6

When I run the migration tool as a dry run:

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys

INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
the bluestore label of osd.0 on node 'horcrux01' carries no 'osd_key', so a rotated key could not
be made to survive a reboot
the bluestore label of osd.6 on node 'horcrux03' carries no 'osd_key', so a rotated key could not
be made to survive a reboot
the bluestore label of osd.12 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.13 on node 'horcrux03' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.14 on node 'horcrux02' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.15 on node 'horcrux01' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.16 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.17 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.18 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.19 on node 'horcrux03' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
and 5 more

So it looks like these OSDs simply never had osd_key written into their bluestore label, which blocks the whole migration. I'd rather not manually recreate 20 OSDs if there's a supported way to fix this.

Questions:

1. Is there a supported way to backfill the osd_key label (e.g. via ceph-bluestore-tool set-label-key) so the migration can proceed?
2. Is this expected for OSDs created via an older pveceph/ceph-volume version, and is there a newer pve-manager/migration script version (no-subscription/test repo) that already handles this case?
3. Since the cluster isn't exposed to any untrusted network, is it safe to just mute these warnings for now and wait for the migration tooling to mature further?

Thanks in advance for any pointers!

 

[ph0x]

Questions:

1. Is there a supported way to backfill the osd_key label (e.g. via ceph-bluestore-tool set-label-key) so the migration can proceed?

That's the way. You can write a simple script getting the device path from ceph-volume and the osd_key from /var/lib/ceph/osd/ceph-X/keyring or from ceph auth get osd.X to set the bluestore value via ceph-bluestore-tool.