Ceph CVEs – keyring rotation strongly advised, probably required in the future

root@pixi0:~# ps aux | grep pvestatd root 5574 0.0 0.0 6528 1952 pts/0 S+ 09:19 0:00 grep pvestatd
but thats the grep command finding its own grep process in ps output, so pvestatd is indeed not running? Could you check on all three nodes

systemctl status pvestatd --no-pager -l

And maybe some older log history as well using e.g. journalctl -u pvestatd -b -n 100 --no-pager or so?

If there are no error messages that would seem like pointing to the cause why starting fails, start the service manually (web UI or systemctl start pvestatd).
 
  • Like
Reactions: Burkhard Linke
There are some hints for that in https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_cephx_how_it_works section, but we did not want to blow the docs up completely.

But roughly, active ceph sessions authenticated with the a ceph monitor and got a monitor ticket from that. This monitor ticket has a (default) TTL of 3 days (but if a monitors old key gets wiped it will expire immediately) and can be used to get ceph service ticket, which are then then used to authenticated with the OSD or MDS (which asks the monitor if its valid) for actual ceph IO. The service ticket has a (default) TTL of 1h.

Upstream ceph has a primitve form of staging a new client key, but that key will be automatically activated once its used, making the old keys obsolete - this was adapted in the Proxmox ceph packages to be more graceful (and we're in the process of upstreaming it, but need to do some upfront work at upstream to make it acceptable there). So, with an external non-Proxmox Ceph setup that has upstream behavior it's likely that once you start the process of changing the client key and moved the first service over, you also started clock for all other services, as their ticket might not get renewed after the TTLs expire.

For PVE you would switch the storage to the new client key, and then to move services over to use it. For VMs its enough to live-migrate them or alternatively restart them, CephFS needs to be remounted and containers on RBD restarted, there is no possibiliity to teach a running librbd or the like of a new client key to allow using either. But this is a code path we did not tested as extensively, or at least not I in such a way that I got all the info correctly available from top of my head, so please test this and read up yourself before doing anything.
And this is the first time the problem of the client key rotation in applications has been explained in a precise and clean manner. Two thumb up for it.
 
exactly.

But the relevant ceph packages are not yet available on the ceph enterprise repository, so how did you got those already installed?

ceph-squid no-subscription
For PVE i have the pve-enterprise subscription.
 
Last edited:
ceph-squid no-subscription
FYI: If you got a valid subscription that allows you accessing pve-enterprise, then you can also use the ceph enterprise repository without having to do anything extra, just add it for all nodes (e.g. in the UI under Node -> Repositories -> Add) and disable the squid no-subscription one.

For the time being you can either wait out until the manager packages come to pve-enterprises or - mentioning for completness sake - temporarily enable the pve-no-subscription repo too, I'd strongly recommend testing the latter in some test lab setup before doing so on in any production environment, just wanted to name all the options. Downgrading ceph would be theoretically an option, but IMO much more involved compared to the other two options.
 
  • Like
Reactions: devaux
FYI: If you got a valid subscription that allows you accessing pve-enterprise, then you can also use the ceph enterprise repository without having to do anything extra, just add it for all nodes (e.g. in the UI under Node -> Repositories -> Add) and disable the squid no-subscription one.

For the time being you can either wait out until the manager packages come to pve-enterprises or - mentioning for completness sake - temporarily enable the pve-no-subscription repo too, I'd strongly recommend testing the latter in some test lab setup before doing so on in any production environment, just wanted to name all the options. Downgrading ceph would be theoretically an option, but IMO much more involved compared to the other two options.
Oh, I didn't know that <o>
Thanks a lot for clarifying!

I also have other clusters currently running with Ceph "no-subscription" 19.2.5. What would be the recommended approach there? Should I switch the Ceph repository to "Ceph Squid Enterprise" before upgrading?

For this specific cluster where I've already updated, I think I will just wait it out until the manager packages land in pve-enterprise. Should I go ahead and switch this cluster to the "Squid Enterprise" Ceph repo right now as well?
 
Last edited:
I also have other clusters currently running with Ceph "no-subscription" 19.2.5. What would be the recommended approach there? Should I switch the Ceph repository to "Ceph Squid Enterprise" before upgrading?
Yes, then it's in sync with the PVE enterprise one, as we will coordinate the rollout of updates between our PVE and ceph enterprise repos.

Should I go ahead and switch this cluster to the "Squid Enterprise" Ceph repo right now as well?
That would be safe to do now and would make sense to ensure it's not overlooked for future updates.