Ceph CVEs – keyring rotation strongly advised, probably required in the future

U

UweSauter

Member
Feb 16, 2024
12
3
8
https://www.openwall.com/lists/oss-security/2026/08/19/4
https://www.openwall.com/lists/oss-security/2026/08/20/20

;tldr There are flaws in the currently used authentication mechanism for CephX. Versions 19.2.6 and 20.2.4 introduce new algorithms to CephX that require a rotation of the keyrings.

What would be the steps to rotate the keyrings on a PVE cluster? Would this require a maintenance of the whole cluster or could it be done gradually?

Even if the old authentication scheme is supported for some coming versions we'll eventually come to the point where existing clusters need to be migrated to the new scheme.

Thanks,

Uwe
 
  • Like
Reactions: kfietz, dakota.marshall and gurubert
I am also curious about what the "official" way to do this is.

It doesnt appear that PVE lets you rotate the keyring from the UI, so it seems to me like the current process is to create a new client keyring with the upgraded crypto, create a new storage pool in PVE with the new keyring, and migrate storage over?? This doesnt seem ideal so I would love to know if there is a better way.
 
UweSauter said:
What would be the steps to rotate the keyrings on a PVE cluster? Would this require a maintenance of the whole cluster or could it be done gradually?
Click to expand...
This can be done gradually, but needs a bit of care. We're currently in the process of finishing a migration helper tool and providing documentation for how to do this.
 
  • Like
Reactions: gurubert, carles89, VictorSTS and 1 other person
There is now a newer pve-manager with version 9.2.12 available on the pve-test repo and a backport of that as 8.4.22 for PVE 8's pvetest repo, matching our respective test repo of the ceph-squid and ceph-tentacle releases.

There is some better handling for this in the service bulk restart endpoint and also with health issues in general, but most notable for this issue is the migration script that needs to be executed once per full cluster after all nodes have been upgraded (doesn't matter on which node). That all is documented in the ceph chapter of the just uploaded pve-docs package with version 9.2.5. That version also contains the documentation for manually migrating over, just in case.

We'd appreciate any feedback here.
 
  • Like
Reactions: jtru, VictorSTS, gurubert and 2 others
Hello Thomas! @t.lamprecht

1787845721234.png 1787847746052.png

After applying the updates in the script its green again, had to use a couple of commands though:

Code: 
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-mon-key --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --wipe-rotating-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-client-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key --apply

The docs worked fine, although the "Rotate the Service Keys by Hand" might confuse beginners as they maybe do not know if they should also use this or not. Why? because after "Rotate the Service Keys by Hand " there are still some steps "Finish the Migration " you also need to do when using the helper. I know that there is already a text saying this, but people might skip the last todos because the manually rotating section is so long, you might not think about, that there is still something left to do to fix the last two messages. Maybe adding a box with "Only use this if you cannot use the helper, if you have used helper before without issues, no need for this part - please go to the section "Finish the Migration "" or something like that?

After that long chapter I still needed to do: "Finish the Migration" to get the last 2 warnings away.

Thanks for the help! Clusters healthy again. But ceph always worked without issues :)
 
Last edited:
I think that it is important to note that there is currently no supported way to migrate the client.osd-lockbox.* cephx keys of existing encrypted OSDs to aes256k — their persistent copy lives outside the auth database, so rotating them would leave the OSD unable to unlock. Recreating the OSD is the only migration path, which is not a reasonable remedy at scale. AUTH_INSECURE_CLIENT_KEY_TYPE and AUTH_INSECURE_KEYS_ALLOWED therefore have to be muted (sticky, with a review date). This affects only pre-upgrade OSDs — new ones are created on the new cipher — and the exposure is limited to a credential already held on the OSD's own node.
 
Thanks for your feedback. There is now a newer pve-manager with version 9.2.14 available on the pve-test repo, together with pve-docs 9.2.6, with the next round of improvements for the cephx migration script, most of them based on the feedback in this thread.

The largest change is that encrypted OSDs are covered now too. The new --rotate-lockbox-keys option rotates the client.osd-lockbox key both in the auth database and in the LVM tag that ceph-volume reads at activation, without stopping any OSD, so recreating the OSD is no longer the only way forward and the advice to mute that warning is gone from the docs.
The options can be combined in a single --apply run, a finished run names the options that are still open, an apply run can be started on any node and a second one simply waits for the first, and interrupted runs (admin key, live key swap, mon. rotation) resume correctly now. The output and the docs chapter were trimmed to what you actually need for the decision. That should hopefully improve UX a bit here and avoid that one has to read that much docs and script output to ensure to get it right.

While we tested this end to end here, feedback from anyone would be still very welcome, especially with more than a handful of encrypted OSDs in the mix.
 
  • Like
Reactions: Sviat, Johannes S and carles89
Where can I find the documentation for this migration tool? I cannot locate the usage instructions.
 
Thank you for your work!

I'm getting this and scratch my head…
Code: 
root@pixi0:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
INFO: This is a dry run. No Ceph key, cipher setting, or daemon will be changed. Session observations can update the migration journal.
FAIL: Could not verify 'aes256k' support for every service daemon:
  could not verify the installed Ceph version of mon. on node 'pixi0'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mon. on node 'pixi1'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mon. on node 'pixi2'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mgr.pixi0 on node 'pixi0'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mgr.pixi1 on node 'pixi1'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mgr.pixi2 on node 'pixi2'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mds.pixi0 on node 'pixi0'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mds.pixi1 on node 'pixi1'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of mds.pixi2 on node 'pixi2'; check that 'pvestatd' runs there
  could not verify the installed Ceph version of osd.0 on node 'pixi0'; check that 'pvestatd' runs there
  and 5 more

root@pixi0:~# ps aux | grep pvestatd
root        5574  0.0  0.0   6528  1952 pts/0    S+   09:19   0:00 grep pvestatd
 
Last edited:
t.lamprecht said:
We further improve the migration procedure and are now continuing with the roll-out, see https://forum.proxmox.com/threads/c...re-and-ceph-19-2-squid-going-eol-soon.186276/

Note that as long as your ceph setup is not accesibly from untrusted networks, the impact here is much reduced and a migration should not be rushed.
Click to expand...
Thanks for the clarification. We use an external ceph cluster, and all services keys have already been rotated. We plan to finished the client keys until the end of the month, so we are not in a hurry.

The exact procedure for the rotating the client keys themselves is easy. But how are active ceph sessions (e.g. kernel mounted RBDs or cephfs mounts) affected? The key rotation does not alter the key itself (this would be a desaster for the lockbox keys), but only changes the representation and encoding of the keys. So the key itself that is used in active sessions should still be valid, and the session should not be interrupted.
Is this assumption correct? In this rotating the client keys should be straightforward. If this is not the case, the method mentioned in comment #2 seems to be the best approach.
 
I ran into this exact issue today, and my cluster is now sitting at HEALTH_ERR.

As I understand it, this is a security warning regarding insecure key types rather than an immediate data loss risk, especially since the cluster isn't publicly exposed.
However, I'm unsure about the correct path forward. I'm running pve-enterprise and can't easily use the migration tools from pve-no-subscription. Will this HEALTH_ERR state block or impact a regular upgrade/migration to tentacle, or should I rotate the keys first to get back to HEALTH_OK?"
 
Last edited:
Not sure whether it will block updates, that might depend on the deployment method. cephadm might be more picky than others.

There's a different problem with having a cluster in HEALTH_ERR for a long time. During this period certain cleanups, garbage collections and other internals are not performed, since the information might be needed for recovery, e.g. older osd maps. I don't know whether this is a selective mechanism (depending on the actual cause), or simply tight to the cluster state itself.
The result might be a grow in mon store size, up to exceeding the local disk capacity. if you did not plan enough headroom in the server configuration, this might become a problem after weeks/months/whatever.
 
Why would you assume cephadm as deployment method? This is specifically for Proxmox installations with Ceph deployed by the mechanism Proxmox provides. I don't think that other deployment methods are supported by the Proxmox software (and team).
 
Burkhard Linke said:
The exact procedure for the rotating the client keys themselves is easy. But how are active ceph sessions (e.g. kernel mounted RBDs or cephfs mounts) affected? The key rotation does not alter the key itself (this would be a desaster for the lockbox keys), but only changes the representation and encoding of the keys. So the key itself that is used in active sessions should still be valid, and the session should not be interrupted.
Is this assump
Click to expand...
There are some hints for that in https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_how_it_works section, but we did not want to blow the docs up completely.

But roughly, active ceph sessions authenticated with the a ceph monitor and got a monitor ticket from that. This monitor ticket has a (default) TTL of 3 days (but if a monitors old key gets wiped it will expire immediately) and can be used to get ceph service ticket, which are then then used to authenticated with the OSD or MDS (which asks the monitor if its valid) for actual ceph IO. The service ticket has a (default) TTL of 1h.

Upstream ceph has a primitve form of staging a new client key, but that key will be automatically activated once its used, making the old keys obsolete - this was adapted in the Proxmox ceph packages to be more graceful (and we're in the process of upstreaming it, but need to do some upfront work at upstream to make it acceptable there). So, with an external non-Proxmox Ceph setup that has upstream behavior it's likely that once you start the process of changing the client key and moved the first service over, you also started clock for all other services, as their ticket might not get renewed after the TTLs expire.

For PVE you would switch the storage to the new client key, and then to move services over to use it. For VMs its enough to live-migrate them or alternatively restart them, CephFS needs to be remounted and containers on RBD restarted, there is no possibiliity to teach a running librbd or the like of a new client key to allow using either. But this is a code path we did not tested as extensively, or at least not I in such a way that I got all the info correctly available from top of my head, so please test this and read up yourself before doing anything.
 
  • Like
Reactions: gurubert
devaux said:
As I understand it, this is a security warning regarding insecure key types rather than an immediate data loss risk, especially since the cluster isn't publicly exposed.
Click to expand...
exactly.
devaux said:
However, I'm unsure about the correct path forward. I'm running pve-enterprise and can't easily use non-standard migration tools. Will this HEALTH_ERR state block or impact a regular upgrade/migration to tentacle, or should I rotate the keys first to get back to HEALTH_OK?"
Click to expand...
But the relevant ceph packages are not yet available on the ceph enterprise repository, so how did you got those already installed?
 
You must log in or register to reply here.
Top Bottom
Back