[SOLVED] CEPH Key Migration now "Not a proper rbd authentication file error"

B

buldamoosh

New Member
Apr 26, 2025
8
0
1
I upgraded my cluster to 9.2.18 and started getting complaints from CEPH about keys. I followed the docs to the letter on CEPH key migration and everything seemed to go well, without errors. Now I can't mount my local CEPH mount.
I've upgraded and now I'm getting: "Not a proper rbd authentication file: /etc/pve/priv/ceph/CEPH00.keyring"
Any help is greatly appreciated as I had migrated all of my VMs/LXCs to my CEPH cluster.

pvesm list CEPH00
Not a proper rbd authentication file: /etc/pve/priv/ceph/CEPH00.keyring

rbd -m 192.168.0.30 -p CEPH00 --keyring /etc/pve/priv/ceph/CEPH00.keyring ls
vm-100-disk-0
vm-101-disk-0
vm-102-disk-0
vm-103-disk-0
vm-104-disk-0
vm-105-disk-0
vm-210-disk-0
vm-210-disk-1
vm-211-disk-0
vm-211-disk-1
vm-300-disk-0
vm-301-disk-0
vm-310-disk-2
vm-400-disk-0
vm-401-disk-0

CEPH File looks like:
[client.admin]
key = KEY
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"

command: ceph auth get client.admin
[client.admin]
key = KEY
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"

What am I missing here?
 
Since the keyring appears to match the output of `ceph auth get client.admin`, another place to double-check might be the `rbd: CEPH00` entry in `/etc/pve/storage.cfg`, especially the `username` value.
 
Thanks for sharing that. I may be missing something, but I don’t see a `username` entry in that RBD storage definition. Maybe that is worth checking against the client name in the keyring.
 
Same here. I upgraded to tentacle a couple weeks ago without any issue. Today i have done a update to 9.2.20, and under ceph i got this:
Cephx key migration checks are active. By themselves, they do not indicate unavailable storage, failed services, or degraded data. Cephx migration guide
 

Attachments

  • Screenshot_2026-09-14 15-01-35.png
    Screenshot_2026-09-14 15-01-35.png
    163.4 KB · Views: 17
@Heracleos The warnings are for you per https://forum.proxmox.com/threads/c...re-and-ceph-19-2-squid-going-eol-soon.186276/. This thread is about a "Not a proper rbd authentication file" error message when following the procedure.

kawano-y said:
Thanks for sharing that. I may be missing something, but I don’t see a `username` entry in that RBD storage definition.
Click to expand...
FWIW our file does not have a username entry for Ceph, only the PBS storage. (have not tried this key rotation process yet)
 
  • Like
Reactions: Heracleos
I had exactly the same error on a 3-node Proxmox VE cluster.

The error was:

Not a proper rbd authentication file: /etc/pve/priv/ceph/pool.keyring

I'm running Proxmox VE 9.2.20.

In my case, I solved it by:
  1. Removing the affected Ceph pool.
  2. Removing /etc/pve/priv/ceph/pool.keyring.
  3. Running apt update && apt upgrade -y on all three nodes.
  4. Rebooting all three nodes.
  5. Creating the Ceph pool again.
After recreating the pool, the RBD storage worked normally again.

My /etc/pve/storage.cfg is:

rbd: cephstorage01
content rootdir,images
krbd 0
pool cephstorage01
I'm not sure whether deleting and recreating the pool was actually necessary, but this was the procedure that fixed the issue for me.
 
I've tried the above - removing the pool, upgrading, rebooting all hosts (5 in my case), and recreating the pool. Nothing - same result. CEPH/RBD storage doesn't mount.

pvesm list CEPH01
Not a proper rbd authentication file: /etc/pve/priv/ceph/CEPH01.keyring

I'm lost.
 
I've tried completely removing my OSDs, the ceph configuration, recreating everything, and restarting all of my nodes....
 
Does apt update && apt full-upgrade -y only work to fix CEPH if you have a valid enterprise subscription?

I'm building a test environment to see if Proxmox and CEPH will work for my company's needs, but I keep running into "Not a proper rbd authentication file: /etc/pve/priv/ceph/ceph-test-01.keyring (500)"
 
I request any thoughts on a very similar problem. After a cephx upgrade to aes256k, both my rbd and cephfs volumes are not available. This is a hyperconverged setup.

Sorry for the dump below, but I figure I should try to provide a reasonable amount of relevant info.

# ceph -s
cluster:​
health: HEALTH_OK​
services:​
mon: 5 daemons, quorum beta,gamma,zeta,delta,epsilon (age 55s)​
mgr: beta(active, since 2h), standbys: delta, gamma, zeta, epsilon​
mds: 1/1 daemons up, 4 standby​
osd: 25 osds: 25 up (since 31s), 25 in (since 8w)​
data:​
volumes: 1/1 healthy​
pools: 8 pools, 321 pgs​
objects: 27.47M objects, 12 TiB​
usage: 36 TiB used, 213 TiB / 248 TiB avail​
pgs: 312 active+clean​

# pvesm status
Not a proper rbd authentication file: /etc/pve/priv/ceph/cbd.keyring​
mount error: Job failed. See "journalctl -xe" for details.​
Name Type Status Total (KiB) Used (KiB) Available (KiB) %​
cbd rbd inactive 0 0 0 0.00%​
cfs cephfs inactive 0 0 0 0.00%​
local dir active 98497780 13713124 79735108 13.92%​
local-lvm lvmthin active 365760512 0 365760512 0.00%​
​
# cat /etc/pve/priv/ceph/cbd.keyring
[client.admin]​
key = AgD...XXX...CDA=​
caps mds = "allow *"​
caps mgr = "allow *"​
caps mon = "allow *"​
caps osd = "allow *"​

# journalctl -xe
░░​
░░ A start job for unit mnt-pve-cfs.mount has begun execution.​
░░​
░░ The job identifier is 35495.​
Sep 19 16:13:44 delta kernel: libceph: auth protocol 'cephx' mauth authentication failed: -13​
Sep 19 16:13:44 delta kernel: libceph: mon0 (1)172.16.11.12:6789 socket closed (con state OPEN)​
Sep 19 16:13:44 delta kernel: libceph: mon0 (1)172.16.11.12:6789 session lost, hunting for new mon​
Sep 19 16:13:44 delta kernel: ceph: No mds server is up or the cluster is laggy​
Sep 19 16:13:44 delta mount[74021]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you >​
Sep 19 16:13:44 delta systemd[1]: mnt-pve-cfs.mount: Mount process exited, code=exited, status=32/n/a​
░░ Subject: Unit process exited​
░░ Defined-By: systemd​
░░​
░░ An n/a= process belonging to unit mnt-pve-cfs.mount has exited.​
░░​
░░ The process' exit code is 'exited' and its exit status is 32.​
Sep 19 16:13:44 delta systemd[1]: mnt-pve-cfs.mount: Failed with result 'exit-code'.​
░░ Subject: Unit failed​
░░ Defined-By: systemd​
░░​
░░ The unit mnt-pve-cfs.mount has entered the 'failed' state with result 'exit-code'.​
Sep 19 16:13:44 delta systemd[1]: Failed to mount mnt-pve-cfs.mount - /mnt/pve/cfs.​
░░ Subject: A start job for unit mnt-pve-cfs.mount has failed​
░░ Defined-By: systemd​
░░​
░░ A start job for unit mnt-pve-cfs.mount has finished with a failure.​
░░​
░░ The job identifier is 35495 and the job result is failed.​

# ceph fs status cfs
cfs - 0 clients​
===​
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS​
0 active zeta Reqs: 0 /s 427k 424k 11.2k 0​
POOL TYPE USED AVAIL​
cfs_metadata metadata 9.85G 1470G​
cfs_data data 34.5T 57.0T​
STANDBY MDS​
beta​
delta​
gamma​
epsilon​

My interpretation of the above is that ceph is happy, but the pve clients can't reach it.

Any help most welcome.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back