Cephx Key Migration Procedure and Ceph 19.2 Squid Going EOL Soon

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
6,890
5,579
315
South Tyrol/Italy
shop.proxmox.com
Recent security findings and fixes in Ceph make it necessary to upgrade ceph and migrate authentication keys from the aes to the aes256k cipher. This is to address weaknesses in the old Cephx authentication method, especially if your Ceph service networks are not isolated. Upgrading to Ceph Tentacle 20.2.4 or newer, or Ceph Squid 19.2.6 or newer, will trigger new Cephx health errors and warnings for the old aes type. These do not by themselves indicate data loss, data corruption, or a service failure, but they flag a security issue that needs to be addressed. See the upstream release notes for more details.

In Proxmox VE, we added a migration script to assist you with this one-time migration of existing setups. We also improved Ceph's key-staging mechanism to allow a smoother, more graceful handover: both keys remain valid while you refresh clients, including running guests and CephFS mounts, to use the new key.

The Ceph packages and accompanying pve-manager and pve-docs updates recently became available in the no-subscription repositories after prolonged internal testing. Given that this is not a trivial migration, we still recommend testing the procedure before rolling it out to production, as our testing cannot cover every setup-specific edge case. We plan to roll these changes out to the enterprise repositories in the second half of next week; depending on further QA and feedback here, this may be delayed by a few days.

The overall procedure is documented in the Proxmox VE Reference Documentation included with your installation and linked from the Ceph dashboard, or online at: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_migration

We recommend following the steps closely and using the migration script as much as possible. With hyper-converged setups and no external clients, the procedure is relatively straightforward, but still requires caution. Check client compatibility first; Proxmox VE's kernel RBD and CephFS clients need a running kernel 7.0 or newer. In particular, do not run commands that retire an old key or restrict the allowed ciphers before every affected client is compatible and has been refreshed. Otherwise, incompatible or not-yet-refreshed clients may see I/O failures on reconnecting or when existing service tickets expire, which can be minutes or days after the change.

For external Ceph users not managed by Proxmox VE, you'll need to manage their key rotation yourself. External clients using a managed Ceph user's key also need to be refreshed, including their saved key copies. See our documentation and the upstream documentation linked there.

Reminder: Ceph 19.2 Squid Going EOL Soon
Please also remember that Ceph 19.2 Squid is estimated to reach its upstream end of life (EOL) on 2026-10-31. If you have not already done so, plan to upgrade existing Ceph Squid setups to Ceph Tentacle while Squid is still supported for a smooth handover. This requires an up-to-date Proxmox VE 9.2 or newer; if you're still on Proxmox VE 8, plan that upgrade first. See the upgrade how-to: https://pve.proxmox.com/wiki/Ceph_Squid_to_Tentacle.

We welcome your feedback!
 
Upgraded yesterday to 9.2.18 and 20.2.4, then ran through the process successfully. Relatively simple 3-node cluster. My only gripe is that in some cases it wasn't 100% clear if I should proceed, since the script didn't have the nice green PASS text for many of the early steps. What is a warning, what is informational, etc. I didn't see outright errors, so proceeded past those points.

Another part that gave me pause was: "After Ceph updates its health status, the two error-severity checks clear. The warning about rotating service keys can remain for a few hours and clears automatically."
It wasn't blatantly obvious whether I needed to wait for that to clear or proceed with the process, without digging a bit deeper into what was actually being done (I proceeded without incident and they cleared some hours later).
 
  • Like
Reactions: alain and findo99
I successfully upgraded a 3-node cluster to ceph version 19.2.6 yesterday as well. I also came across a couple of passages that were unclear to me, so I wasn't sure whether it was safe to proceed , some parts read as somewhat ambiguous. I do have to admit I was caught off guard by having to deal with this in the middle of the cluster upgrade, which probably made the docs harder to follow than they would have been otherwise.
 
Last edited:
Upgraded yesterday to 9.2.18 and 20.2.4, then ran through the process successfully. Relatively simple 3-node cluster. My only gripe is that in some cases it wasn't 100% clear if I should proceed, since the script didn't have the nice green PASS text for many of the early steps. What is a warning, what is informational, etc. I didn't see outright errors, so proceeded past those points.

Another part that gave me pause was: "After Ceph updates its health status, the two error-severity checks clear. The warning about rotating service keys can remain for a few hours and clears automatically."
It wasn't blatantly obvious whether I needed to wait for that to clear or proceed with the process, without digging a bit deeper into what was actually being done (I proceeded without incident and they cleared some hours later).
thanks for the feedback, I tried to improve on that in newer pve-manager (currently on pve-test).
 
I've upgraded and now I'm getting: Not a proper rbd authentication file: /etc/pve/priv/ceph/CEPH00.keyring
Any help is greatly appreciated as I had migrated all of my VMs/LXCs to my CEPH cluster.
CEPH File looks like:
[client.admin]
key = KEY
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
command: ceph auth get client.admin
[client.admin]
key = KEY
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
What am I missing here?