Accidentally ran ceph auth rm client.admin from one of my monitor nodes

[DemiNe0]

Hi Everyone,

I Accidentally ran `ceph auth rm client.admin` from one of my monitor nodes. I was following a tutorial for adding ceph to k8s and misunderstood one of the steps on the tutorial.

Anytime I try to run a command from any of the nodes now I get the following error:
monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)

Is there anyway I can recover this? It doesn't appear that proxmox can connect to ceph anymore.

 

[abma]

just a guess, maybe:

# ceph auth import -i /etc/pve/priv/ceph/cephrbd.keyring

helps?

 

[DemiNe0]

just a guess, maybe:

# ceph auth import -i /etc/pve/priv/ceph/cephrbd.keyring

helps?

It doesn't. I get the same
-1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster) error

 

[abma]

https://stackoverflow.com/questions/51316722/ceph-delete-client-admin-accidently:

ceph -n mon. --keyring /var/lib/ceph/mon/ceph-node-1/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'

maybe? ("ceph-node-1" needs to be adjusted)

 

[DemiNe0]

I've tried that as well, although I get a different error:

root@pve02:/var/lib/ceph/mon/ceph-pve02# ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 auth: unable to find a keyring on /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 AuthRegistry(0x7fb77405ad68) no keyring found at /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool <poolstr> [<states>...]
pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
pg ls [<pool:int>] [<states>...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>]
Error EINVAL: invalid command

I've tried different takes on -n mon. as well:

root@pve02:/var/lib/ceph/mon/ceph-pve02# history | grep "ceph -n "
  242  ceph -n mon. --keyring keyring  auth caps client.admin mds 'allow *' osd 'allow *' mon 'allow *'
  249  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  250  ceph -n mon.ceph-pve02 --keyring /var/lib/ceph/mon/ceph-pve02/keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  253  ceph -n mon.ceph-pve02 --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  254  ceph -n client.admin.keyring --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  255  ceph -n client.admin --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  256  ceph -n mon. --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  257  ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  258  bash -c "ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'"
  294  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'

 

[abma]

without a test environment i can't help it seems, sorry: but i'm really interested as this is sth. which could happen everyone i guess.

did you try to ask on the ceph mailing list or irc? https://ceph.io/en/community/connect/

as i guess "ceph status" already fails, this is more a ceph problem than a proxmox problem.

 

[abma]

from the docs, this seems very relevant:

https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/

maybe disabling cephx does the trick.

 

[DemiNe0]

Ya, I tried posting to the users mailing list, however it doesn't appear that the message has made it through. I'm registered properly on there. Not sure why my message won't appear.

I'll try disabling cephx and recreating the client.admin that way.

 

[DemiNe0]

I disabled cephx and recreated the client.admin token. I then copied the token into the existing client.admin keyring and copied that to the other servers.

My VM's that have drives on ceph are able to launch and read the data from them now. I assume this is because of cephx being off. Proxmox still cannot view the ceph datastores from the UI or do any migrations or the like with anything on ceph.

Task viewer: VM 112 - Clone
create full clone of drive virtio0 (CephBlk:vm-112-disk-0)
TASK ERROR: clone failed: rbd error: rbd: couldn't connect to the cluster!

or

rbd error: rbd: listing images failed: (95) Operation not supported (500)

 

[abma]

so does "ceph status" work?

IMHO the next step should be, to reenable cephx, check if it works (ceph status, etc) and then try to fix the communication with pve again which basicly should be only

https://pve.proxmox.com/pve-docs/pveceph.1.html#_ceph_client

cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring


"my-ceph-storage" needs to be adjusted to your system it seems.

 

[DemiNe0]

Alright, That problem was caused by having cephx disabled while at the same time having storage keys at /etc/pve/priv/ceph/
I removed the storage keys at /etc/pve/priv/ceph/ and that fixed the issue.

I had tried copying the new admin keyring over the ceph storage keyring however that didn't work. I'm probably just going to reinstall ceph. It's easy enough now that I can migrate the vm's off that storage temporarily.