Accidentally ran ceph auth rm client.admin from one of my monitor nodes

DemiNe0

Member
Oct 19, 2017
24
7
23
36
Hi Everyone,

I Accidentally ran `ceph auth rm client.admin` from one of my monitor nodes. I was following a tutorial for adding ceph to k8s and misunderstood one of the steps on the tutorial.

Anytime I try to run a command from any of the nodes now I get the following error:
monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)

Is there anyway I can recover this? It doesn't appear that proxmox can connect to ceph anymore.
 

DemiNe0

Member
Oct 19, 2017
24
7
23
36
just a guess, maybe:

# ceph auth import -i /etc/pve/priv/ceph/cephrbd.keyring

helps?
It doesn't. I get the same
-1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster) error
 

DemiNe0

Member
Oct 19, 2017
24
7
23
36
I've tried that as well, although I get a different error:
Code:
root@pve02:/var/lib/ceph/mon/ceph-pve02# ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 auth: unable to find a keyring on /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 AuthRegistry(0x7fb77405ad68) no keyring found at /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool <poolstr> [<states>...]
pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
pg ls [<pool:int>] [<states>...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>]
Error EINVAL: invalid command

I've tried different takes on -n mon. as well:

Code:
root@pve02:/var/lib/ceph/mon/ceph-pve02# history | grep "ceph -n "
  242  ceph -n mon. --keyring keyring  auth caps client.admin mds 'allow *' osd 'allow *' mon 'allow *'
  249  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  250  ceph -n mon.ceph-pve02 --keyring /var/lib/ceph/mon/ceph-pve02/keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  253  ceph -n mon.ceph-pve02 --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  254  ceph -n client.admin.keyring --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  255  ceph -n client.admin --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  256  ceph -n mon. --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  257  ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  258  bash -c "ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'"
  294  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
 

DemiNe0

Member
Oct 19, 2017
24
7
23
36
Ya, I tried posting to the users mailing list, however it doesn't appear that the message has made it through. I'm registered properly on there. Not sure why my message won't appear.

I'll try disabling cephx and recreating the client.admin that way.
 

DemiNe0

Member
Oct 19, 2017
24
7
23
36
I disabled cephx and recreated the client.admin token. I then copied the token into the existing client.admin keyring and copied that to the other servers.

My VM's that have drives on ceph are able to launch and read the data from them now. I assume this is because of cephx being off. Proxmox still cannot view the ceph datastores from the UI or do any migrations or the like with anything on ceph.

Code:
Task viewer: VM 112 - Clone
create full clone of drive virtio0 (CephBlk:vm-112-disk-0)
TASK ERROR: clone failed: rbd error: rbd: couldn't connect to the cluster!
or
Code:
rbd error: rbd: listing images failed: (95) Operation not supported (500)
 
Last edited:

DemiNe0

Member
Oct 19, 2017
24
7
23
36
Alright, That problem was caused by having cephx disabled while at the same time having storage keys at /etc/pve/priv/ceph/
I removed the storage keys at /etc/pve/priv/ceph/ and that fixed the issue.

I had tried copying the new admin keyring over the ceph storage keyring however that didn't work. I'm probably just going to reinstall ceph. It's easy enough now that I can migrate the vm's off that storage temporarily.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!