Ceph CVEs – keyring rotation strongly advised, probably required in the future

[UweSauter]

https://www.openwall.com/lists/oss-security/2026/08/19/4
https://www.openwall.com/lists/oss-security/2026/08/20/20

;tldr There are flaws in the currently used authentication mechanism for CephX. Versions 19.2.6 and 20.2.4 introduce new algorithms to CephX that require a rotation of the keyrings.

What would be the steps to rotate the keyrings on a PVE cluster? Would this require a maintenance of the whole cluster or could it be done gradually?

Even if the old authentication scheme is supported for some coming versions we'll eventually come to the point where existing clusters need to be migrated to the new scheme.

Thanks,

Uwe

 

[dakota.marshall]

I am also curious about what the "official" way to do this is.

It doesnt appear that PVE lets you rotate the keyring from the UI, so it seems to me like the current process is to create a new client keyring with the upgraded crypto, create a new storage pool in PVE with the new keyring, and migrate storage over?? This doesnt seem ideal so I would love to know if there is a better way.

 

[t.lamprecht]

What would be the steps to rotate the keyrings on a PVE cluster? Would this require a maintenance of the whole cluster or could it be done gradually?

This can be done gradually, but needs a bit of care. We're currently in the process of finishing a migration helper tool and providing documentation for how to do this.

 

[t.lamprecht]

There is now a newer pve-manager with version 9.2.12 available on the pve-test repo and a backport of that as 8.4.22 for PVE 8's pvetest repo, matching our respective test repo of the ceph-squid and ceph-tentacle releases.

There is some better handling for this in the service bulk restart endpoint and also with health issues in general, but most notable for this issue is the migration script that needs to be executed once per full cluster after all nodes have been upgraded (doesn't matter on which node). That all is documented in the ceph chapter of the just uploaded pve-docs package with version 9.2.5. That version also contains the documentation for manually migrating over, just in case.

We'd appreciate any feedback here.

 

[jsterr]

Hello Thomas! @t.lamprecht



After applying the updates in the script its green again, had to use a couple of commands though:

root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-mon-key --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --wipe-rotating-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-client-keys --apply
root@training3:~# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key --apply

The docs worked fine, although the "Rotate the Service Keys by Hand" might confuse beginners as they maybe do not know if they should also use this or not. Why? because after "Rotate the Service Keys by Hand " there are still some steps "Finish the Migration " you also need to do when using the helper. I know that there is already a text saying this, but people might skip the last todos because the manually rotating section is so long, you might not think about, that there is still something left to do to fix the last two messages. Maybe adding a box with "Only use this if you cannot use the helper, if you have used helper before without issues, no need for this part - please go to the section "Finish the Migration "" or something like that?

After that long chapter I still needed to do: "Finish the Migration" to get the last 2 warnings away.

Thanks for the help! Clusters healthy again. But ceph always worked without issues

 

[iss-integration]

I think that it is important to note that there is currently no supported way to migrate the client.osd-lockbox.* cephx keys of existing encrypted OSDs to aes256k — their persistent copy lives outside the auth database, so rotating them would leave the OSD unable to unlock. Recreating the OSD is the only migration path, which is not a reasonable remedy at scale. AUTH_INSECURE_CLIENT_KEY_TYPE and AUTH_INSECURE_KEYS_ALLOWED therefore have to be muted (sticky, with a review date). This affects only pre-upgrade OSDs — new ones are created on the new cipher — and the exposure is limited to a credential already held on the OSD's own node.