Procedure for cycling CEPH keyrings cluster-wide

[xcezz]

Hello,

I want to cycle / renew all CEPH keyrings across the cluster as part of my security maintenance procedures.

My environment

• Proxmox VE 8.2.8
• CEPH 18.2.4
• Components where I want to cycle the keyrings
  • MON & client.admin
  • MGR
  • MDS

Current situation

I tried to rotate the keys in the following order: MON and client.admin keyrings first, then MGR, then MDS and lastly any client keyrings.

The process for each component:

• Create new keyring via ceph-authtool
• Backup old keyring
• Set correct permissions for the newly created keyring
• Copy new keyring to /var/lib/ceph/<component>/ceph-<nodename>/keyring
• Import keyring into DB via ceph auth import

Unfortunately, these steps were not sufficient and I got a "permission denied" error (respectively handle_auth_bad_method). So I had to switch back to my old keyrings to recover.

Questions

• What is the correct order to cycle these keyrings?
• Any dependency between components I'm missing?

Any guidance, especially from those who have performed this operation before, would be greatly appreciated.

Thanks in advance!

 

[gurubert]

IMHO this question should be asked on the Ceph mailing list.

 

[xcezz]

I thought of that too. Still, it seems to me that Proxmox plays a significant role here and maybe the procedure differs slightly compared to an unmanaged Ceph cluster.

Also upgraded Ceph to 19.2.0 Squid (latest) to try to make use of ceph auth rotate, but the command somehow does not exist in either the proxmox ceph no-subscription or the test repository.