Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon

@SteveITS
- Install is up to date: PVE 9.2.20 / Linux 7.0.14-17-pve
- "add --verbose" --> I do not understand
- I ran "--rotate-admin-key --apply"

After reading the manual once again, I sopped-started/live-migrated all VMs, then I rebooted all HVs.
Now it shows me something different:
Code: 
root@hv1|~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Ready for confirmation: client.admin
Confirm only after refreshing every consumer, including disconnected ones and external key copies:
  /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed --restrict-ciphers

root@hv1|~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --confirm-all-clients-refreshed  --restrict-ciphers
INFO: Recovery and confirmations may run before the remaining plan can be shown.

Apply this run, including recovery and any requested key confirmations? (y/N) y
INFO: Collecting cluster info.
WARN: accepting '--confirm-all-clients-refreshed' for client.admin: every consumer of them that this run can see is refreshed, and you confirm the same for those it cannot see
INFO: committing the staged key of 'client.admin', its previous key stops working now
PASS: 'client.admin' now uses the 'aes256k' cipher
INFO: Nothing in this plan is stopped, so the cluster health does not gate this run.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.
FAIL: refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified. Run this without options to check the remaining steps.

Ok, looks were getting closer:
"not every monitor answered the session query"

The rest looks good so far...
 
SteveITS said:
--verbose is an option. Maybe it will show more info.

Are you doing this as root?
Click to expand...

--verbose shows some more info before, but the same error after execution:
- I am root on console but not root in the WebUI.

Code: 
root@hv4:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphers --verbose
INFO: Collecting cluster info.

Where this cluster stands
Ceph reports no service key on the old 'aes' cipher.
The health check cannot inspect the shared 'mon.' key.

Plan
Step 1: allow only the 'aes256k' cipher for authentication, which clears the remaining insecure-key warnings. A key or client on the old cipher is refused from then on. This causes brief monitor elections.
  both settings are stored in the monitor map, so monitors briefly stop answering during each election
  ceph mon set auth_preferred_cipher aes256k
  ceph mon set auth_allowed_ciphers aes256k

'auth_preferred_cipher' (currently 'aes') is left untouched. It decides the cipher of keys created later; on 'aes', new client keys stay usable by kernel clients that do not know 'aes256k'.

/etc/pve/priv/cephx-key-migration.json records migration progress and the pre-rotation key for every key this run changes.

Apply this run, including recovery and any requested key confirmations? (y/N) y
FAIL: refusing to restrict the allowed ciphers: not every monitor answered the session query, so live consumers cannot be verified. Run this without options to check the remaining steps.
root@hv4:~#

++tried this without success:
Z

[SOLVED] Thread 'pve-cephx-rotate-service-keys error in restrict-ciphers phase "not every monitor answered the session query"'

Hi, I'm following the instructions to make the aes -> aes256k ceph key migration.
I've a 3 node cluster, with all the updates installed.
Everything went fine until I had to run the last step:

Bash: 
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
PASS: Every service key uses 'aes256k', and so do the service tickets.

Remaining migration steps:
  --restrict-ciphers: allow only the 'aes256k' cipher for authentication, once every running consumer...
 
Just bumping this thread

I realize my issue (osd_key completely missing from the bluestore label, not just an old/stale value) is a bit different from what's been discussed later in the thread. No worries if nobody's run into this exact case, just wanted to bring it back up in case anyone's had an idea in the meantime.

If it helps, I'm happy to test a fix on a single isolated OSD before rolling anything out cluster-wide, and I can provide more logs/diagnostics if needed.

Thanks in advance to anyone who has a bit of time for this, and sorry if this isn't quite the right format, this is my very first post here @SteveITS
I've tried the verbose but it doesn't add much :

root@horcrux01:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --verbose
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
INFO: ignoring 'osd.10' on node 'horcrux04', 'osd.11' on node 'horcrux04', 'osd.20' on node 'horcrux03', 'osd.23' on node 'horcrux02', 'osd.25' on node 'horcrux01', 'osd.7' on node 'horcrux03', 'osd.8' on nod
e 'horcrux03', 'osd.9' on node 'horcrux04': removed or destroyed daemons, only their data directories are left behind
INFO: collecting daemon keyrings and bluestore labels from 4 node(s)
FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
the bluestore label of osd.0 on node 'horcrux01' carries no 'osd_key', so a rotated key could not
be made to survive a reboot
the bluestore label of osd.6 on node 'horcrux03' carries no 'osd_key', so a rotated key could not
be made to survive a reboot
the bluestore label of osd.12 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.13 on node 'horcrux03' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.14 on node 'horcrux02' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.15 on node 'horcrux01' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.16 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.17 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.18 on node 'horcrux04' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
the bluestore label of osd.19 on node 'horcrux03' carries no 'osd_key', so a rotated key could
not be made to survive a reboot
and 5 more
Click to expand...
 
t.lamprecht said:
Recent security findings and fixes in Ceph make it necessary to upgrade ceph and migrate authentication keys from the aes to the aes256k cipher. This is to address weaknesses in the old Cephx authentication method, especially if your Ceph service networks are not isolated. Upgrading to Ceph Tentacle 20.2.4 or newer, or Ceph Squid 19.2.6 or newer, will trigger new Cephx health errors and warnings for the old aes type. These do not by themselves indicate data loss, data corruption, or a service failure, but they flag a security issue that needs to be addressed. See the upstream release notes for more details.

In Proxmox VE, we added a migration script to assist you with this one-time migration of existing setups. We also improved Ceph's key-staging mechanism to allow a smoother, more graceful handover: both keys remain valid while you refresh clients, including running guests and CephFS mounts, to use the new key.

The Ceph packages and accompanying pve-manager and pve-docs updates recently became available in the no-subscription repositories after prolonged internal testing. Given that this is not a trivial migration, we still recommend testing the procedure before rolling it out to production, as our testing cannot cover every setup-specific edge case. We plan to roll these changes out to the enterprise repositories in the second half of next week; depending on further QA and feedback here, this may be delayed by a few days.

The overall procedure is documented in the Proxmox VE Reference Documentation included with your installation and linked from the Ceph dashboard, or online at: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_migration

We recommend following the steps closely and using the migration script as much as possible. With hyper-converged setups and no external clients, the procedure is relatively straightforward, but still requires caution. Check client compatibility first; Proxmox VE's kernel RBD and CephFS clients need a running kernel 7.0 or newer. In particular, do not run commands that retire an old key or restrict the allowed ciphers before every affected client is compatible and has been refreshed. Otherwise, incompatible or not-yet-refreshed clients may see I/O failures on reconnecting or when existing service tickets expire, which can be minutes or days after the change.

For external Ceph users not managed by Proxmox VE, you'll need to manage their key rotation yourself. External clients using a managed Ceph user's key also need to be refreshed, including their saved key copies. See our documentation and the upstream documentation linked there.

Reminder: Ceph 19.2 Squid Going EOL Soon
Please also remember that Ceph 19.2 Squid is estimated to reach its upstream end of life (EOL) on 2026-10-31. If you have not already done so, plan to upgrade existing Ceph Squid setups to Ceph Tentacle while Squid is still supported for a smooth handover. This requires an up-to-date Proxmox VE 9.2 or newer; if you're still on Proxmox VE 8, plan that upgrade first. See the upgrade how-to: https://pve.proxmox.com/wiki/Ceph_Squid_to_Tentacle.

We welcome your feedback!
Click to expand...


Hi,

Unless I'm mistaken the current https://pve.proxmox.com/wiki/Ceph_Squid_to_Tentacle page doesn't mention any of the above

Sincerely,

Laurent
 
I maintain a still have a productive proxmox-cluster with ceph (with active license) with proxmox 8.4.21 which I wanted to upgrade this week and afraid that this aes->aes256 migration issue will hit me during upgrade as the apt full-upgrade will also upgrade ceph squid to 19.2.6.

I'm unsure how to proceed. Should I try to pin ceph version to 19.2.3 so this will not bring the ceph cluster into the HEALTH_ERROR situation during the proxmox upgrade so I can resolve the ceph-autentication key migration separately.

What do you think? I would prefer not to break the production cluster o_O
 
Technically speaking (in my opinion) the upgrades (to latest 19 or 20) are unrelated to this key rotation issue. Though it might be good to link to the instructions in case people upgrade from older Ceph 19 to 20.

You can just ignore the warnings for the key rotation…in fact you’re doing that now because you’re not seeing them yet. It doesn’t prevent the upgrade.

Note PVE 8 is past EOL and Ceph 19 is EOL next month.
 
Exactly. This is the reason I want to upgrade from proxmox 8 to 9.
But I have upgraded a test proxmox cluster a few weeks ago where I had tested the upgrade process to proxmox 9 and it now offered the ceph upgrade from 19.2.3 to 19.2.6 which enforces the aes->aes25k upgrade with a ceph HEALTH_ERR status.

I have done this aes migration now and I would feel much more comfortable if I could do the proxmox 9 upgrad and the aes25k migration in two separate steps.

My concern is, that during the upgrade of proxmox from 8 to 9 there will be the upgrade ceph 19.2.6 packages included.
 
wech said:
the ceph upgrade from 19.2.3 to 19.2.6 which enforces the aes->aes25k upgrade with a ceph HEALTH_ERR status.
Click to expand...

the package upgrade does not enforce anything. if you upgrade to 19.2.6 ceph will just start to complain/warn about the insecure keys. doing the key rotation is an extra step *after* upgrading.
 
I updated yesterday to 9.2.20 and Ceph to 19.2.6. The migration process ran smoothly without any failures. The colors for the pass and warning messages helped a lot. Thanks for the effort to improve the documentation and the scripts!
 
  • Like
Reactions: wech
fabian said:
the package upgrade does not enforce anything. if you upgrade to 19.2.6 ceph will just start to complain/warn about the insecure keys. doing the key rotation is an extra step *after* upgrading.
Click to expand...
True. Sorry for the bad wording.

I wante to say it complains (with a pretty drastic looking) HEALTH_ERROR after upgrading ceph to 19.2.6, but the ceph-health-warnings can be muted for days or weeks. So in reality it really does not "enforce" the key migration, but it still looks unsettling when a ceph warning exists during upgrading of the proxmox nodes. I would prefer to have a healthy cluster while upgrading the nodes and it seems I missed this opporunity :-(

I was just wondering if it would be possible or a bad idea to pin ceph to 19.2.3 so I can do the proxmox upgrade with a healthy looking ceph cluster and after finishing that, remove ceph pinning and work on the key migration separately.
 
If you want to not rotate keys now (i.e., not see the warning) you can just hide or ignore the messages. You're trying to avoid seeing a warning that doesn't affect Ceph functionality/operations. The only reason you're not seeing the warning now is the warning/check for it doesn't exist in older versions of Ceph.

In other words, yes, you can upgrade PVE now and rotate the Ceph keys next week.

Or just upgrade PVE to the latest 8.x, rotate keys in Ceph, and upgrade PVE to 9.x next week.
 
SteveITS said:
If you want to not rotate keys now (i.e., not see the warning) you can just hide or ignore the messages. You're trying to avoid seeing a warning that doesn't affect Ceph functionality/operations. The only reason you're not seeing the warning now is the warning/check for it doesn't exist in older versions of Ceph.

In other words, yes, you can upgrade PVE now and rotate the Ceph keys next week.

Or just upgrade PVE to the latest 8.x, rotate keys in Ceph, and upgrade PVE to 9.x next week.
Click to expand...
Thanks. I think this muting the messages is probably really the way to go.

Upgrading ceph to 19.2.6 on PVE 8.4 is not possible, I guess because it is EOL. It does not offer the ceph upgrade to 19.2.6, latest available ceph version there is 19.2.3 - just checked again.
 
just went through the key migration this morning. it was a sudden unexpected surprise after doing a simple apt upgrade from 9.2.10 to 9.2.20 (which included the newer 19.2.6 ceph i think) after rebooting around half my cluster i saw the health_err regarding the key migration and was caught off guard. im assuming a bunch of others doing an update from an earlier 9.x or 8.x->9.2.20 is also going to be in for the same surprise. maybe not the best way to go about the key migration from a user-friendly point of view, so i hope some admins see this post/thread and save themselves a shock.

that aside, i calmed down a bit and went through the migration and everything went fine. i admit the wiki/documentation/script isn't worded the best, it's a bit confusing a times about what you should be doing after running/re-running the script but i got the hang of it after a while.

if it helps anyone: basically run the first script in the documentation without the '--apply' and if the message at the bottom comes back green (saying preflight is ok) then you should be safe to run same script with the '--apply'. then depending on your cluster/etc you will have different results. after you handle whatever it mentions, re-run same script again without '--apply' and if green again (preflight ok) run with '--apply' and basically repeat until nothing else remains. then move onto next script and follow same pattern of dry-run vs 'apply' run. the most confusing part was regarding the admin key...basically once EVERYTHING is ready, the script will actually print on the screen the next command to run (script regarding the confirm-all-clients-refreshed) and give another green light message with it. if the script doesnt specifically printout the 'confirm-all-clients-refreshed' script in it's results, then you have something else lingering to fix and/or wait for.
for the most part my experinece was
1. run the 'migrate cluster keys' script...dryrun then apply
2. it did most of the migrate/rotating services
3. usually the thing i got stuck on for this section was having to remove ISOs that were mounted in my vms CD-ROMs from my cephFS iso storage. after removing them all i reran script (dry run then apply) and that got me the green light for the next section.
3.5. this is where the red health_err go away and you are left with warnings. if you have a warning for the 'rotating service keys' i just let mine sit there for about an hour and they cleared themselves. i personally waited for the rotating keys warnings to go away before i did the very final script in step 7, but from what i read maybe you can mute/ignore that one? im not entirely sure if safe or not, i hope they can elaborate and clarify the instructions a bit better. but that's just what i did and waiting for them to disappear worked totally fine.
4. section for 'migrate keys for compatible ceph users'. ran dryrun first then apply.
5. it did some stuff, then got stuck on a few things (listing potential 'kernel sockets' in my case meant live-migrate your VMs off that node to another)
6. once i migrated all my VMs around to other nodes i reran script again as dryrun/apply (you can go from node1 -> node2 even if node2 is listed in your script...just also move node2 VMs to node1 or node3 or whatever...just as long as the VMs migrate once you should be good.)
7. this is when the script finally added an additional line that included the next script to run (the script for confirm-all-clients)
8. i ran the script it provided, and it worked fine. after that, everything was done, errors/warnings gone...just had to unset noout. all good.

hope this helps a little bit on what to expect. aside from running the script over and over and removing ISOs from CD-ROMs and migrating VMs from node->other node, i didn't have to do anything else crazy. just keep fixing things until that final script pops up for you, i wasnt expecting it to give me the next script to run (again i hope documentation improves) but if you know to look for it, it might save you from running a script too early because you are uncertain what to do.
 
  • Like
Reactions: wech
croRF said:
this is when the script finally added an additional line that included the next script to run (the script for confirm-all-clients)
Click to expand...
That's basically where I got stuck...it had seemingly ignored the WARN entries up until that point, so an additional one line output about "cannot continue until all WARN issues are resolved" would be user friendly, especially for those doing it during a late night maintenance period :rolleyes: and/or stressed about seeing these "new" warnings.

Otherwise, very helpful scripting/solution, thanks.
 
  • Like
Reactions: wech
I very much appreciate croRF's post. The docs are indeed crude and I cannot get this wrong. I am waiting a bit longer for the documentation to get clarified a bit more and to upgrade a bit further into the future on the enterprise platform as newer upgrades become available.

Another question, when I query the ceph version, PM responds with 20.2.4. Is this different than 20.2.4-pve3 referenced in the PM docs? Do I have the correct ceph package or is there another, standard, non-PM based ceph package that somehow got installed incorrectly?

Thanks!
 
It looks more complicated than it is. Run the first command and the script output will tell you the next one to run. Until the last one you can just stop/pause.

the -pve suffix indicates patches by Proxmox. If you look at the updates tab in the GUI (as root) you can see pending updates. If you don’t already see the keys warning in Ceph in the web GUI you’re not on the latest.
 
for the ones with radosgw for S3 if you get 403 when trying to upload new files, temporary disable rgw_sigv4

Bash: 
node-01:~# ceph config get client rgw_sigv4_insecure
false
node-01:~# ceph config set client rgw_sigv4_insecure true
node-01:~# ceph config get client rgw_sigv4_insecure
true

ref: CVE-2026-54330
 
You must log in or register to reply here.
Top Bottom
Back