How to restore cephx auth on a auth free ceph cluster?

[blackpaw]

Have a Ceph Cluster setup using Proxmox, where the authentication was disabled to try and improve performance (not noticeably), this was some time ago.

/etc/pve/ceph.conf

[global]
         auth_client_required = none
         auth_cluster_required = none
         auth_service_required = none

I somewhat naively tried just setting auth to cephx which result in a non-functional cluster and a stressful evening for me Next time I'm moving the virtual router to local storage first.

I presume I have to re-setup the keys and secrets first, any proxmox guides to this? I know the locations are different to a std ceph setup.

Thanks - Lindsay

nb. Want to restore auth because we need to access ceph rbd from a Nomad cluster using ceph-csi (same as Kubernetes) and you don't seem to be able to disable auth in its client.

 

[blackpaw]

Well I managed to figure it out - mostly.

2 of the mon's were missing the ceph.mon.keyring and the other one had an out of date copy. On each node running a monitor I:

cp /etc/pve/priv/ceph.mon.keyring  /var/lib/ceph/mon/ceph-a/keyring (ceph-a altered for the name of your node)

Then restart all mons. mgs and osd. Could then do a ceph status etc. Cluster is now do a *slow* Global Recovery Event, but is healthy.

The only thing that isn't workin is that proxmox can't list the vm images in the storage Vm list, it displays:
rbd error: rbd: listing images failed: (2) No such file or directory (500)

rbd ls works fine from the command line and syslog shows no error. VM's are running fine.

 

[blackpaw]

rbd error: rbd: listing images failed: (2) No such file or directory (500)

Fixed that too - there was a duff image in the vm list from a canceled disk move. Deleting that fixed the listing.