Accidentally ran ceph auth rm client.admin from one of my monitor nodes

DemiNe0

Active Member
Oct 19, 2017
28
10
43
38
Hi Everyone,

I Accidentally ran `ceph auth rm client.admin` from one of my monitor nodes. I was following a tutorial for adding ceph to k8s and misunderstood one of the steps on the tutorial.

Anytime I try to run a command from any of the nodes now I get the following error:
monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)

Is there anyway I can recover this? It doesn't appear that proxmox can connect to ceph anymore.
 
just a guess, maybe:

# ceph auth import -i /etc/pve/priv/ceph/cephrbd.keyring

helps?
It doesn't. I get the same
-1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster) error
 
I've tried that as well, although I get a different error:
Code:
root@pve02:/var/lib/ceph/mon/ceph-pve02# ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 auth: unable to find a keyring on /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2021-10-27T17:06:59.288+0000 7fb77b16b700 -1 AuthRegistry(0x7fb77405ad68) no keyring found at /etc/ceph/ceph.mon..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool <poolstr> [<states>...]
pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
pg ls [<pool:int>] [<states>...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [<threshold:int>]
Error EINVAL: invalid command

I've tried different takes on -n mon. as well:

Code:
root@pve02:/var/lib/ceph/mon/ceph-pve02# history | grep "ceph -n "
  242  ceph -n mon. --keyring keyring  auth caps client.admin mds 'allow *' osd 'allow *' mon 'allow *'
  249  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  250  ceph -n mon.ceph-pve02 --keyring /var/lib/ceph/mon/ceph-pve02/keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  253  ceph -n mon.ceph-pve02 --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  254  ceph -n client.admin.keyring --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  255  ceph -n client.admin --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  256  ceph -n mon. --keyring keyring  get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  257  ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
  258  bash -c "ceph -n mon. --keyring keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'"
  294  ceph -n mon. --keyring /var/lib/ceph/mon/ceph-pve02/keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *'
 
Ya, I tried posting to the users mailing list, however it doesn't appear that the message has made it through. I'm registered properly on there. Not sure why my message won't appear.

I'll try disabling cephx and recreating the client.admin that way.
 
I disabled cephx and recreated the client.admin token. I then copied the token into the existing client.admin keyring and copied that to the other servers.

My VM's that have drives on ceph are able to launch and read the data from them now. I assume this is because of cephx being off. Proxmox still cannot view the ceph datastores from the UI or do any migrations or the like with anything on ceph.

Code:
Task viewer: VM 112 - Clone
create full clone of drive virtio0 (CephBlk:vm-112-disk-0)
TASK ERROR: clone failed: rbd error: rbd: couldn't connect to the cluster!
or
Code:
rbd error: rbd: listing images failed: (95) Operation not supported (500)
 
Last edited:
Alright, That problem was caused by having cephx disabled while at the same time having storage keys at /etc/pve/priv/ceph/
I removed the storage keys at /etc/pve/priv/ceph/ and that fixed the issue.

I had tried copying the new admin keyring over the ceph storage keyring however that didn't work. I'm probably just going to reinstall ceph. It's easy enough now that I can migrate the vm's off that storage temporarily.
 
Same issue here.
I was trying to mont cephfs on a VM.

After ...
Code:
mkdir /etc/ceph
ssh root@<node-ip> "sudo ceph config generate-minimal-conf" | sudo tee /etc/ceph/ceph.conf

... I tried to get a keyring to authenticate client.admin to access the cepph cluster with
Code:
ssh root@<node-ip> "sudo ceph fs authorize cephfs client.admin / rw" | sudo tee /etc/ceph/ceph.client.admin.keyring

That one led to the error message that this user is already there and cannot created again followed by the advice to remove it prior creating it anew.

With ...
Code:
ssh root@<node-ip> "sudo ceph auth rm client.admin"
followed by
Code:
ssh root@<node-ip> "sudo ceph fs authorize cephfs client.admin / rw" | sudo tee /etc/ceph/ceph.client.admin.keyring

I removed the already existing client.admin, got an new client.admin and the desired keyring - but then ceph was messed up as described above.

Questions:
- How can I retrieve the client-admin keyring to be used to authenticate a cephfs mount?
- What exactly needs to be done with the new admin keyring created to make the cluster work again?
(As a possible solution if somebody falls in the same trap again ... Oo )

Fatzit:
At the end I had to remove the whole ceph installation from the cluster and reinstall it from scratch with running something like this on every node ...

--- Attention ---
---
--- This will probably be only the last resort prior reinstalling the whole cluster from scratch !!!
--- Dont experiment with such things on production systems if you dont have proper backups of all relevent data !!!

---
Code:
rm -rf /etc/systemd/system/ceph*
killall -9 ceph-mon ceph-mgr ceph-mds
rm -rf /var/lib/ceph/mon/  /var/lib/ceph/mgr/  /var/lib/ceph/mds/
pveceph purge
apt purge ceph-mon ceph-osd ceph-mgr ceph-mds
rm /etc/init.d/ceph
pveceph install

... to solve that carelessly self induced problem, just because I did not had the knowledge to fix such a bad thing. Nor do I have it today.

Conclusions:
Well ... Just don't tamper with any kind of "admin" users at all - nowhere - never - ever.
Especially if you don't have the understanding of the consequences....

Create your own users instead.
e.g.
Code:
ssh root@<node-ip> "sudo ceph fs authorize cephfs client.stephan / rw" | sudo tee /etc/ceph/ceph.client.stephan.keyring

You can use that on with ceph-fuse or fstab as well.

- Manual ceph-fuse Mount
Code:
ceph-fuse --id stephan -k /etc/ceph/ceph.client.stephan.keyring /mnt/cephfs
or
Code:
ceph-fuse --id stephan -k /etc/ceph/ceph.client.stephan.keyring --client_mds_namespace your-cephfs /mnt/your-cephfs
if you have more than one fs set up on the cluster

- fstab ceph-fuse Mount
Code:
client_mountpoint=/,id=stephan    /mnt/cephfs    fuse.ceph    defaults,_netdevnoatime    0    0
Code:
client_mountpoint=/,client_fs=cephfs,id=stephan    /mnt/cephfs    fuse.ceph    defaults,_netdev,noatime    0    0

- Manual kernel Mount
Code:
mount -t ceph <mon-ip>[,<monip>]:/ /mnt/cephfs -o name=stephan

- fstab kernel Mount
Code:
<mon-ip>[,<monip>]:/    /mnt/cephfs    ceph   name=stephan,_netdev,noatime    0    0

By now I did not find out how to kernel mount a non default cephfs with fstab, but I guess there is an option to be passed as well.
Make sure that you use the _netdev option in any case to mount the fs after networking has been completed.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!