Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon

fiona said:
Hi @egan wu,
do you still have entries for those manager services in your /etc/pve/ceph.conf? Do they show up in the UI in a node's Ceph > Monitor panel?
Click to expand...
I'm having the same problem. A node had a root fs failure and was removed some time ago. Keys of it's mds and mgr are causing a FAIL with --rotate-cluster-keys. The removed node is not visible in UI, but grep -r still finds it from /var/lib/ceph/mon/*/store.db and there is still directory and some files for it in /etc/pve/nodes.

How do I delete keys of a removed node?
 
mijutu said:
I'm having the same problem. A node had a root fs failure and was removed some time ago. Keys of it's mds and mgr are causing a FAIL with --rotate-cluster-keys. The removed node is not visible in UI, but grep -r still finds it from /var/lib/ceph/mon/*/store.db and there is still directory and some files for it in /etc/pve/nodes.

How do I delete keys of a removed node?
Click to expand...
You could try : "ceph auth del {TYPE}.{ID}"
 
  • Like
Reactions: mijutu and fiona
Th.Naumann said:
All new ceph-keys (type aes256k) are valid and present in files on clusternodes - checked with "ceph auth ls" and "cat /var/lib/ceph/osd|mon|mgr/*/keyring. After reboot of any clusternode with osds the old osd-key (type aes) are restored in local keyring-files "/var/lib/ceph/osd/*/", so start of osds fail. All other keys are untouched. After getting correct key for osd.X "ceph auth get osd.X > /var/lib/ceph/osd/ceph-X/keyring" and restarting osds everything works fine.
Click to expand...

Hi there.

Quote from the docs:
An OSD keyring lives on a tmpfs that ceph-volume rebuilds on every boot,priming it from the osd_key entry in the bluestore label of the OSD’s blockdevice. Writing only the keyring file therefore works until the next reboot,after which the OSD comes back with its old key and can no longer authenticate.
Click to expand...

You can see the stored key with ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block .

If you see your old key then the section "Rotate the Service Keys by Hand" from the pve docs should fix this.

I reccomend reading the section in the documentation but here are the commands I used per OSD:
Code: 
systemctl stop ceph-osd@0
ceph osd down 0
ceph auth rotate --key-type=aes256k osd.0 -o /root/osd.0.keyring
KEY=$(ceph-authtool /root/osd.0.keyring -n osd.0 --print-key)
ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-0/block --key osd_key --value "$KEY"
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
ceph-bluestore-tool prime-osd-dir --dev /var/lib/ceph/osd/ceph-0/block --path /var/lib/ceph/osd/ceph-0 --no-mon-config
chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
systemctl start ceph-osd@0

Hope this helps.
 
woodstock said:
Hi there.

Quote from the docs:


You can see the stored key with ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block .

If you see your old key then the section "Rotate the Service Keys by Hand" from the pve docs should fix this.

I reccomend reading the section in the documentation but here are the commands I used per OSD:
Code: 
systemctl stop ceph-osd@0
ceph osd down 0
ceph auth rotate --key-type=aes256k osd.0 -o /root/osd.0.keyring
KEY=$(ceph-authtool /root/osd.0.keyring -n osd.0 --print-key)
ceph-bluestore-tool set-label-key --dev /var/lib/ceph/osd/ceph-0/block --key osd_key --value "$KEY"
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block
ceph-bluestore-tool prime-osd-dir --dev /var/lib/ceph/osd/ceph-0/block --path /var/lib/ceph/osd/ceph-0 --no-mon-config
chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
systemctl start ceph-osd@0

Hope this helps.
Click to expand...
Hi woodstock,

the old aes-keys for osds were in the label under /var/lib/ceph/osd/*/block. Setting the actual aes256k-keys via bluestore-tool solved the problem.
Thank you very much for your hint and support!
 
An update:
t.lamprecht said:
The Ceph packages and accompanying pve-manager and pve-docs updates recently became available in the no-subscription repositories after prolonged internal testing. Given that this is not a trivial migration, we still recommend testing the procedure before rolling it out to production, as our testing cannot cover every setup-specific edge case. We plan to roll these changes out to the enterprise repositories in the second half of next week; depending on further QA and feedback here, this may be delayed by a few days.
Click to expand...
The Ceph packages and accompanying pve-manager and pve-docs updates are now available in the Proxmox VE 9 enterprise repositories as well.
 
  • Like
Reactions: devaux, chughes, VictorSTS and 2 others
Hi everyone,

I upgraded from Proxmox VE 8 to 9 without being aware that this release came with breaking changes around Ceph (I completely missed the CephX key migration requirement). I've since tried searching the docs and asking an LLM for help, but I haven't found a working solution.

My cluster is currently in HEALTH_WARN with the following checks:
  • 8 auth client entities with insecure key types
  • Monitors are configured to allow auth using insecure key types
  • Monitors are configured to allow creation of insecure key types
  • 4 rotating auth service keys using insecure key types
  • 24 auth service entities with insecure key types
  • Monitors are configured to issue insecure service tickets
  • 2 OSD(s) experiencing slow operations in BlueStore
Ceph version: 19.2.6, 4 nodes (horcrux01–04), 20 OSDs total (all up/in), 385 PGs active+clean.

When I run the migration tool as a dry run:

Bash: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys

INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  the bluestore label of osd.0 on node 'horcrux01' carries no 'osd_key', so a rotated key could not
    be made to survive a reboot
  the bluestore label of osd.6 on node 'horcrux03' carries no 'osd_key', so a rotated key could not
    be made to survive a reboot
  the bluestore label of osd.12 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.13 on node 'horcrux03' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.14 on node 'horcrux02' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.15 on node 'horcrux01' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.16 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.17 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.18 on node 'horcrux04' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  the bluestore label of osd.19 on node 'horcrux03' carries no 'osd_key', so a rotated key could
    not be made to survive a reboot
  and 5 more

So it looks like these OSDs simply never had osd_key written into their bluestore label, which blocks the whole migration. I'd rather not manually recreate 20 OSDs if there's a supported way to fix this.

Questions:
  1. Is there a supported way to backfill the osd_key label (e.g. via ceph-bluestore-tool set-label-key) so the migration can proceed?
  2. Is this expected for OSDs created via an older pveceph/ceph-volume version, and is there a newer pve-manager/migration script version (no-subscription/test repo) that already handles this case?
  3. Since the cluster isn't exposed to any untrusted network, is it safe to just mute these warnings for now and wait for the migration tooling to mature further?
Thanks in advance for any pointers!
 
How can I manually rotate keys like keys for rbd-mirror?


Code: 
root@pve-3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.

Not touched by this run:
  Additional actions not selected: cipher restriction.

Plan
Step 1: rotate 1 selected Ceph user key and rewrite every copy Proxmox VE keeps. Affected CephFS mounts are redone on every node unless something is using them.
  Ceph user 'client.admin': staged next to the current key
  For each staged Ceph user key, both the current and new keys authenticate until the new key is committed with '--confirm-clients-refreshed USER' or, once every open record is ready, '--confirm-all-clients-refreshed'.
  Live sessions of the selected Ceph users:
  'client.admin', 7 sessions: pve-1: 2, pve-2: 2, pve-3: 3
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
  pve-1: 2 unattributed sockets (possible kernel clients); rbd-mirror (3)
  pve-2: 2 unattributed sockets (possible kernel clients)
  pve-3: VM 102; unattributed socket (possible kernel client)
  For staged keys, refresh their consumers after applying: live-migrate every VM, remount every kernel CephFS or RBD mount, and restart other consumers; a dry run then reports who is left.

Left to whoever manages the client that reads them; 'man pveceph' covers what each needs:
  client.rbd-mirror-peer-a
  client.rbd-mirror.pve-1

I thought there was a chapter about manually rotating keys, but I cant find it in the docs anymore.
 
jsterr said:
How can I manually rotate keys like keys for rbd-mirror?


Code: 
root@pve-3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.

Not touched by this run:
  Additional actions not selected: cipher restriction.

Plan
Step 1: rotate 1 selected Ceph user key and rewrite every copy Proxmox VE keeps. Affected CephFS mounts are redone on every node unless something is using them.
  Ceph user 'client.admin': staged next to the current key
  For each staged Ceph user key, both the current and new keys authenticate until the new key is committed with '--confirm-clients-refreshed USER' or, once every open record is ready, '--confirm-all-clients-refreshed'.
  Live sessions of the selected Ceph users:
  'client.admin', 7 sessions: pve-1: 2, pve-2: 2, pve-3: 3
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
  pve-1: 2 unattributed sockets (possible kernel clients); rbd-mirror (3)
  pve-2: 2 unattributed sockets (possible kernel clients)
  pve-3: VM 102; unattributed socket (possible kernel client)
  For staged keys, refresh their consumers after applying: live-migrate every VM, remount every kernel CephFS or RBD mount, and restart other consumers; a dry run then reports who is left.

Left to whoever manages the client that reads them; 'man pveceph' covers what each needs:
  client.rbd-mirror-peer-a
  client.rbd-mirror.pve-1

I thought there was a chapter about manually rotating keys, but I cant find it in the docs anymore.
Click to expand...
In official ceph-docu you can find something about that topic:

https://docs.ceph.com/en/latest/rados/operations/health-checks/
 
Rebuilt my entire proxmox pve cluster (6-node) with 9.2.2 and after creating a ceph pool, get this error right away "Not a proper rbd authentication file: /etc/pve/priv/ceph/pool05.keyring"
 
bryanem32 said:
Rebuilt my entire proxmox pve cluster (6-node) with 9.2.2 and after creating a ceph pool, get this error right away "Not a proper rbd authentication file: /etc/pve/priv/ceph/pool05.keyring"
Click to expand...
this should be addressed by adding the ceph-client packages to the regular pve repositories - are any package updates still open? (apt update; apt full-upgrade)
 
For me it ran straight forward, but theres one issue left:
Code: 
HEALTH_WARN: 1 auth client entities with insecure key types
Use the migration helper for cluster-owned keys and keys of compatible Ceph users. Leave user keys required by incompatible consumers unchanged.

entity client.admin using insecure key type: aes

What I did:
Code: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-mon-key --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --wipe-rotating-keys --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-client-keys --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key --apply
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --apply

Check:
Code: 
root@pve2:~# ceph auth get client.admin
[client.admin]
        key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
        pending key = YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY=
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

Any Suggestions?
 
Last edited:
  • Like
Reactions: mijutu
Hi all,

I'm running the cephx key migration (aes → aes256k) following the official procedure in the pveceph docs, and it fails on Step 1 (rotating the shared mon. key), before touching anything else. Looking for confirmation on the safest way to proceed, since I don't have a support subscription.

Environment

pve-manager: 9.2.20
ceph: 20.2.4-pve4 (all 9 nodes)
3 monitors (in quorum), 3 mgr, 3 mds, 48 OSD
Cluster used exclusively through Proxmox (no external Ceph clients)
pveceph auth status before running: service: 54 aes, client: 8 aes, pending keys: 0, monitor quorum reports aes256k capable: yes

What I ran

/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --apply

Output

INFO: Collecting cluster info.
INFO: collecting daemon keyrings and bluestore labels from 9 node(s)
...
Apply this run, including recovery and any requested key confirmations? (y/N) y

Rotating the shared monitor key
FAIL: error with 'auth get': mon_cmd failed - failed to find mon. in keyring

Post-failure verification (cluster looked healthy, same HEALTH_ERR as before, quorum intact, no new warnings, pending keys: 0):

ceph -s
# quorum: 3/3 mons, 48/48 osd up, 257 active+clean, HEALTH_ERR unchanged (same 2 cipher-related checks as before the run)

ceph auth get mon.
Error ENOENT: failed to find mon. in keyring

ceph auth ls | grep -A3 '^mon\.'
# (no output — mon. entity does not appear in the auth database at all)

cat /etc/pve/priv/ceph.mon.keyring
[mon.]
key = <REDACTED>
caps mon = "allow *"
[client.admin]
key = <REDACTED>
...

cat /var/lib/ceph/mon/ceph-<hostname>/keyring
[mon.]
key = <REDACTED — same value as above>
caps mon = "allow *"

So the mon. key exists and is consistent across the Proxmox-managed copy and each monitor's local keyring, but it's genuinely absent from the cluster's internal auth database (ceph auth ls doesn't list it, ceph auth get mon. returns ENOENT).

What I found researching this: the upstream Ceph docs (auth-config-ref, "Upgrading and Rotating CephX Keys" section) confirm this is expected historical behavior — "The mon. historically has not been managed by the Monitor auth database; it exists solely in each Monitor's keyring inside its data directory." The documented upstream fix is:

ceph auth rotate --key-type=aes256k mon. | tee mon.keyring

This apparently both creates the entity in the auth db and rotates it to aes256k in one step.

My concern before doing this manually: the pveceph docs explicitly warn that the upstream procedure "does not account for these Proxmox VE details" and that /etc/pve/priv/ceph.mon.keyring must always be merged, never overwritten, since it's the authoritative copy pveceph mon create uses when (re)initializing a monitor. If I run the upstream ceph auth rotate command directly, the Ceph auth db gets the new key, but Proxmox's own managed keyring copy would still have the old one — which seems like it would break any future monitor recreation.

Questions for the community / Proxmox team:

  1. Is this mon. missing from the auth db a known gap in pve-cephx-rotate-service-keys, or is my cluster's bootstrap history unusual in some way?
  2. Is there a Proxmox-aware way to resolve this (ideally something the migration helper itself should handle), or is manually running ceph auth rotate --key-type=aes256k mon. followed by manually merging the new key into /etc/pve/priv/ceph.mon.keyring considered safe?
  3. Has anyone hit this exact error and found a fix, or is there a newer pve-manager build in the pipeline that handles it?

I also asked AI about how to fix it and got another approach:

1789759164686.png

I don't have an active support subscription, so any guidance here is appreciated before I touch anything mon-related on a live 3-node quorum. Happy to provide more diagnostics if useful.

Thanks!
 
Last edited:
erickmelo said:
FAIL: error with 'auth get': mon_cmd failed - failed to find mon. in keyring
Click to expand...
FWIW I did not see this error but also "ceph auth ls | grep -A3 '^mon\.'" shows no output.

@t.lamprecht
The docs say, "After checking disconnected clients and external copies, run the exact command the dry run prints. When every rotation is ready and no key needs the old cipher, it will print the options --apply --confirm-all-clients-refreshed --restrict-ciphers."

However I was never given that command with parameter "--restrict-ciphers"...?

The last one was:
Code: 
# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --rotate-cluster-keys
INFO: Collecting cluster info.
WARN: No OSD in this cluster carries the fsid of client.osd-lockbox.02c40964-8cb9-(...23 more of these entries...)-4043-97db-f7bf52dfdb43, so these lockbox keys are left alone. Remove one only after confirming the OSD was destroyed and no node carries its ceph-volume device; absence from the OSD map alone is not enough.
PASS: Every service key uses 'aes256k', and so do the service tickets.

my command history:
Code: 
  502  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys
  503  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --apply
  504  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key
  505  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-all-storage-keys --rotate-admin-key --apply
(migrated all VMs here, via PVE maintenance mode)
  514  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
  515  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --rotate-cluster-keys
  516  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed
  517  pveceph auth status
  518  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --rotate-cluster-keys

"pveceph auth status" has "Keys needing attention (24 listed identities)" with the client.osd-lockbox entries noted above, as "current aes (old), pending none."

Listed current keys (Ceph identities, not guests, mounts, or sessions)
source: auth dump-keys
service: 31 aes256k
client: 24 aes, 33 aes256k
pending keys: 0

I am thinking the "24" are all old, somehow? The 33 are expected/correct. I think it is waiting for the 24 to be "fixed"...can they be removed somehow?

Edit: without counting through records, it's quite possible these are all old OSDs. Does the PVE Destroy not remove these entries?
 
Last edited:
Hi,

you can run the helper script "/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys" without options to check what remains.
 
  • Like
Reactions: jsterr
it says the following:

Code: 
root@hv2:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Client keys awaiting action:
  'client.admin': 81 session(s) still authenticate with a previous key (hv1: 18, hv2: 20, hv3:
    24, hv4: 19). Both keys remain valid. Refresh these consumers, then rerun without options.
Possible consumers (host-wide hints, not session attribution):
Kernel hints mean sockets without an owning process.
  hv1: VM 100, VM 102, VM 106, VM 107, VM 113, VM 114, VM 115, VM 121, 3 more VMs; unattributed socket (possible kernel client)
  hv2: VM 108, VM 117, VM 120, VM 123, VM 143, VM 144, VM 145, VM 148, 5 more VMs; unattributed socket (possible kernel client)
  hv3: VM 101, VM 103, VM 104, VM 105, VM 116, VM 118, VM 122, VM 124, 7 more VMs; unattributed socket (possible kernel client)
  hv4: VM 129, VM 131, VM 133, VM 135, VM 137, VM 139, VM 140, VM 142, 2 more VMs; unattributed socket (possible kernel client)
 
You must log in or register to reply here.
Top Bottom
Back