Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon

I stumbled over this thread and was wondering why I am not seeing this warning. Turns out the enterprise repo doesn't seem to have 19.2.5 -- is that on purpose? I understand that 19.2.6 isn't there yet, but 19.2.5 is already old?

I have this in my sources:
Code: 
Types: deb
URIs: https://enterprise.proxmox.com/debian/ceph-squid
Suites: trixie
Components: enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

and apt-cache policy ceph says:
Code: 
ceph:
  Installed: 19.2.3-pve4
  Candidate: 19.2.3-pve4
  Version table:
 *** 19.2.3-pve4 500
        500 https://enterprise.proxmox.com/debian/ceph-squid trixie/enterprise amd64 Packages
        100 /var/lib/dpkg/status

...

EDIT:// So if I had to guess it is probably the following: 19.2.4 & 19.2.5 weren't critical enough to put into the enterprise repository and 19.2.6 is still awaiting further QA (as written in the first post). Is there any reason to wait for 19.2.6 and fix the aes issues there before upgrading to Tentacle or is upgrading to Tentacle now and fixing it in Tentacle once it hits the enterprise repos just as good?
 
Last edited:
fiona said:
Hi @egan wu,
do you still have entries for those manager services in your /etc/pve/ceph.conf? Do they show up in the UI in a node's Ceph > Monitor panel?
Click to expand...
I'm having the same problem. A node had a root fs failure and was removed some time ago. Keys of it's mds and mgr are causing a FAIL with --rotate-cluster-keys. The removed node is not visible in UI, but grep -r still finds it from /var/lib/ceph/mon/*/store.db and there is still directory and some files for it in /etc/pve/nodes.

How do I delete keys of a removed node?
 
mijutu said:
I'm having the same problem. A node had a root fs failure and was removed some time ago. Keys of it's mds and mgr are causing a FAIL with --rotate-cluster-keys. The removed node is not visible in UI, but grep -r still finds it from /var/lib/ceph/mon/*/store.db and there is still directory and some files for it in /etc/pve/nodes.

How do I delete keys of a removed node?
Click to expand...
You could try : "ceph auth del {TYPE}.{ID}"
 
  • Like
Reactions: fiona
Th.Naumann said:
All new ceph-keys (type aes256k) are valid and present in files on clusternodes - checked with "ceph auth ls" and "cat /var/lib/ceph/osd|mon|mgr/*/keyring. After reboot of any clusternode with osds the old osd-key (type aes) are restored in local keyring-files "/var/lib/ceph/osd/*/", so start of osds fail. All other keys are untouched. After getting correct key for osd.X "ceph auth get osd.X > /var/lib/ceph/osd/ceph-X/keyring" and restarting osds everything works fine.
Click to expand...

Hi there.

Quote from the docs:
An OSD keyring lives on a tmpfs that ceph-volume rebuilds on every boot,priming it from the osd_key entry in the bluestore label of the OSD’s blockdevice. Writing only the keyring file therefore works until the next reboot,after which the OSD comes back with its old key and can no longer authenticate.
Click to expand...

You can see the stored key with ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block .

If you see your old key then the section "Rotate the Service Keys by Hand" from the pve docs should fix this.

I reccomend reading the section in the documentation but here are the commands I used per OSD:
Code: 
systemctl stop ceph-osd@0
ceph osd down 0
ceph auth rotate --key-type=aes256k osd.0 -o /root/osd.0.keyring
KEY=$(ceph-authtool /root/osd.0.keyring -n osd.0 --print-key)
ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-0/block --key osd_key --value "$KEY"
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
ceph-bluestore-tool prime-osd-dir --dev /var/lib/ceph/osd/ceph-0/block --path /var/lib/ceph/osd/ceph-0 --no-mon-config
chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
systemctl start ceph-osd@0

Hope this helps.
 
woodstock said:
Hi there.

Quote from the docs:


You can see the stored key with ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block .

If you see your old key then the section "Rotate the Service Keys by Hand" from the pve docs should fix this.

I reccomend reading the section in the documentation but here are the commands I used per OSD:
Code: 
systemctl stop ceph-osd@0
ceph osd down 0
ceph auth rotate --key-type=aes256k osd.0 -o /root/osd.0.keyring
KEY=$(ceph-authtool /root/osd.0.keyring -n osd.0 --print-key)
ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-0/block --key osd_key --value "$KEY"
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
ceph-bluestore-tool prime-osd-dir --dev /var/lib/ceph/osd/ceph-0/block --path /var/lib/ceph/osd/ceph-0 --no-mon-config
chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
systemctl start ceph-osd@0

Hope this helps.
Click to expand...
Hi woodstock,

the old aes-keys for osds were in the label under /var/lib/ceph/osd/*/block. Setting the actual aes256k-keys via bluestore-tool solved the problem.
Thank you very much for your hint and support!
 
An update:
t.lamprecht said:
The Ceph packages and accompanying pve-manager and pve-docs updates recently became available in the no-subscription repositories after prolonged internal testing. Given that this is not a trivial migration, we still recommend testing the procedure before rolling it out to production, as our testing cannot cover every setup-specific edge case. We plan to roll these changes out to the enterprise repositories in the second half of next week; depending on further QA and feedback here, this may be delayed by a few days.
Click to expand...
The Ceph packages and accompanying pve-manager and pve-docs updates are now available in the Proxmox VE 9 enterprise repositories as well.
 
  • Like
Reactions: chughes, VictorSTS, woodstock and 1 other person
Hi everyone,

I upgraded from Proxmox VE 8 to 9 without being aware that this release came with breaking changes around Ceph (I completely missed the CephX key migration requirement). I've since tried searching the docs and asking an LLM for help, but I haven't found a working solution.

My cluster is currently in HEALTH_WARN with the following checks:
  • 8 auth client entities with insecure key types
  • Monitors are configured to allow auth using insecure key types
  • Monitors are configured to allow creation of insecure key types
  • 4 rotating auth service keys using insecure key types
  • 24 auth service entities with insecure key types
  • Monitors are configured to issue insecure service tickets
  • 2 OSD(s) experiencing slow operations in BlueStore
Ceph version: 19.2.6, 4 nodes (horcrux01–04), 20 OSDs total (all up/in), 385 PGs active+clean.

When I run the migration tool as a dry run:

Bash: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys

INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  the bluestore label of osd.0 on node 'horcrux01' carries no 'osd_key', so a rotated key could not
    be made to survive a reboot
  the bluestore label of osd.6 on node 'horcrux03' carries no 'osd_key', so a rotated key could not
    be made to survive a reboot
  the bluestore label of osd.12 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.13 on node 'horcrux03' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.14 on node 'horcrux02' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.15 on node 'horcrux01' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.16 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.17 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.18 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.19 on node 'horcrux03' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  and 5 more

So it looks like these OSDs simply never had osd_key written into their bluestore label, which blocks the whole migration. I'd rather not manually recreate 20 OSDs if there's a supported way to fix this.

Questions:
  1. Is there a supported way to backfill the osd_key label (e.g. via ceph-bluestore-tool set-label-key) so the migration can proceed?
  2. Is this expected for OSDs created via an older pveceph/ceph-volume version, and is there a newer pve-manager/migration script version (no-subscription/test repo) that already handles this case?
  3. Since the cluster isn't exposed to any untrusted network, is it safe to just mute these warnings for now and wait for the migration tooling to mature further?
Thanks in advance for any pointers!
 
How can I manually rotate keys like keys for rbd-mirror?


Code: 
root@pve-3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.

Not touched by this run:
  Additional actions not selected: cipher restriction.

Plan
Step 1: rotate 1 selected Ceph user key and rewrite every copy Proxmox VE keeps. Affected CephFS mounts are redone on every node unless something is using them.
  Ceph user 'client.admin': staged next to the current key
  For each staged Ceph user key, both the current and new keys authenticate until the new key is committed with '--confirm-clients-refreshed USER' or, once every open record is ready, '--confirm-all-clients-refreshed'.
  Live sessions of the selected Ceph users:
  'client.admin', 7 sessions: pve-1: 2, pve-2: 2, pve-3: 3
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
  pve-1: 2 unattributed sockets (possible kernel clients); rbd-mirror (3)
  pve-2: 2 unattributed sockets (possible kernel clients)
  pve-3: VM 102; unattributed socket (possible kernel client)
  For staged keys, refresh their consumers after applying: live-migrate every VM, remount every kernel CephFS or RBD mount, and restart other consumers; a dry run then reports who is left.

Left to whoever manages the client that reads them; 'man pveceph' covers what each needs:
  client.rbd-mirror-peer-a
  client.rbd-mirror.pve-1

I thought there was a chapter about manually rotating keys, but I cant find it in the docs anymore.
 
jsterr said:
How can I manually rotate keys like keys for rbd-mirror?


Code: 
root@pve-3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.

Not touched by this run:
  Additional actions not selected: cipher restriction.

Plan
Step 1: rotate 1 selected Ceph user key and rewrite every copy Proxmox VE keeps. Affected CephFS mounts are redone on every node unless something is using them.
  Ceph user 'client.admin': staged next to the current key
  For each staged Ceph user key, both the current and new keys authenticate until the new key is committed with '--confirm-clients-refreshed USER' or, once every open record is ready, '--confirm-all-clients-refreshed'.
  Live sessions of the selected Ceph users:
  'client.admin', 7 sessions: pve-1: 2, pve-2: 2, pve-3: 3
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
  pve-1: 2 unattributed sockets (possible kernel clients); rbd-mirror (3)
  pve-2: 2 unattributed sockets (possible kernel clients)
  pve-3: VM 102; unattributed socket (possible kernel client)
  For staged keys, refresh their consumers after applying: live-migrate every VM, remount every kernel CephFS or RBD mount, and restart other consumers; a dry run then reports who is left.

Left to whoever manages the client that reads them; 'man pveceph' covers what each needs:
  client.rbd-mirror-peer-a
  client.rbd-mirror.pve-1

I thought there was a chapter about manually rotating keys, but I cant find it in the docs anymore.
Click to expand...
In official ceph-docu you can find something about that topic:

https://docs.ceph.com/en/latest/rados/operations/health-checks/
 
Rebuilt my entire proxmox pve cluster (6-node) with 9.2.2 and after creating a ceph pool, get this error right away "Not a proper rbd authentication file: /etc/pve/priv/ceph/pool05.keyring"
 
bryanem32 said:
Rebuilt my entire proxmox pve cluster (6-node) with 9.2.2 and after creating a ceph pool, get this error right away "Not a proper rbd authentication file: /etc/pve/priv/ceph/pool05.keyring"
Click to expand...
this should be addressed by adding the ceph-client packages to the regular pve repositories - are any package updates still open? (apt update; apt full-upgrade)
 
You must log in or register to reply here.
Top Bottom
Back